Advanced Deep Learning with Keras

[ 333 ]
Chapter 10
The training strategy of A2C is different in the sense that it computes gradients from
the last step to the first step. Hence, the return accumulates beginning from the last
step reward or the last next state value:
# the memory is visited in reverse as shown
# in Algorithm 10.5.1
for item in self.memory[::-1]:
[step, state, next_state, reward, done] = item
# compute the return
r = reward + gamma*r
item = [step, state, next_state, r, done]
# train per step
# a2c reward has been discounted
self.train(item)
The reward variable in the list is also replaced by return. It is initialized by reward
if the terminal state is reached (that is, the car touches the flag) or the next state value
for non-terminal states:
v = 0 if reward > 0 else agent.value(next_state)[0]
In the Keras implementation, all the routines that we mentioned are implemented
as methods in the PolicyAgent class. The role of the PolicyAgent is to represent
the agent implementing policy gradient methods including building and training the
network models and predicting the action, log probability, entropy, and state value.
Following listing shows how one episode unfolds when the agent executes and trains
the policy and value models. The for loop is executed for 1000 episodes. An episode
terminates upon reaching 1000 steps or when the car touches the flag. The agent
executes the action predicted by the policy at every step. After each episode or step,
the training routine is called.
Listing 10.6.6, policygradient-car-10.1.1.py: The agent runs for 1000 episodes
to execute the action predicted by the policy at every step and perform training:
# sampling and fitting
for episode in range(episode_count):
state = env.reset()
# state is car [position, speed]
state = np.reshape(state, [1, state_dim])
# reset all variables and memory before the start of
# every episode
step = 0
total_reward = 0
done = False
agent.reset_memory()