Advanced Deep Learning with Keras

Chapter 9
Where all terms are familiar from the previous discussion on Q-Learning and Q(a|s)
→ Q(s,a). The term max Q( s′ , a′ ) → max Q( a′ s′
). In other words, using the Q-Network,
a′ a′
predict the Q value of each action given next state and get the maximum among
max Q a′ | s′ = max Q s′ | a′
= 0 .
them. Note that at the terminal state s', ( ) ( )
Algorithm 9.6.1, DQN algorithm:
Require: Initialize replay memory D to capacity N
a′ a′
Require: Initialize action-value function Q with random weights θ
Require: Initialize target action-value function Q target
with weights θ
Require: Exploration rate, ε and discount factor, γ
1. for episode = 1, …,M do:
2. Given initial state s
3. for step = 1,…, T do:
⎧
sample( a)
random ε⎫
<
a = ⎪
⎨
⎪
4. Choose action argmax Q( s, a;
θ)
otherwise
⎬
⎪⎩
a
⎪⎭
5. Execute action a, observe reward r and next state s'
6. Store transition (s, a, r, s ' ) in D
7. Update the state, s = s '
8. //experience replay
9. Sample a mini batch of episode experiences (s j
, a j
, r j+1
, s j+1
) from D
⎧
rj+
1
if episodeterminates at j + 1⎫
Qmax
= ⎨
⎪
−
10. rj+ 1
γ max Qtarget ( s
j+ 1, a
j+
1;
θ ) otherwise ⎬
⎪
+
⎪
a
⎩
j+
1
⎪⎭
11. Perform gradient descent step on ( Q ( , ; )) max
−Q s
j
a
j
θ −
with respect to
parameters θ
12. // periodic update of the target network
13. Every C steps Q target
= Q, that is set θ = θ
14. End
However, it turns out that training the Q-Network is unstable. There are two
problems causing the instability:
1. A high correlation between samples
2. A non-stationary target
− =
θ
[ 295 ]