Advanced Deep Learning with Keras

mean, stddev = args
dist = tf.distributions.Normal(loc=mean, scale=stddev)
entropy = dist.entropy()
return entropy
The entropy model is only used by the A2C method:
Chapter 10
Figure 10.6.8: A value model
Preceding figure shows the value model. The model also uses the pre-trained
encoder with frozen weights to implement following equation which is repeated
here for convenience:
( , ) ( )
T
t t
θv φ
t
θv
v = V s = s
(Equation 10.3.2)
θ
v
are the weights of the Dense(1) layer, the only layer that receives value gradient
updates. Figure 10.6.8 represents V ( st
, θ
v ) in Algorithms 10.3.1 to 10.5.1. The value
model can be built in a few lines:
inputs = Input(shape=(self.state_dim, ), name='state')
self.encoder.trainable = False
x = self.encoder(inputs)
value = Dense(1,
activation='linear',
kernel_initializer='zero',
name='value')(x)
self.value_model = Model(inputs, value, name='value')
These lines are also implemented in method build_actor_critic(), which is
shown in Listing 10.6.2.
[ 327 ]