Advanced Deep Learning with Keras

Policy Gradient Methods
Figure 10.6.7: Entropy model
Apart from the policy network, π( at
st
, θ ), we must also have the action log probability
(logp) network In π( at
| st
, θ ) since this is actually what calculates the gradient. As
shown in Figure 10.6.6, the logp network is simply the policy network where an
additional Lambda(1) layer computes the log probability of the Gaussian distribution
given action, mean, and standard deviation. The logp network and actor (policy)
model share the same set of parameters. The Lambda layer does not have any
parameter. It is implemented by the following function:
# given mean, stddev, and action compute
# the log probability of the Gaussian distribution
def logp(self, args):
mean, stddev, action = args
dist = tf.distributions.Normal(loc=mean, scale=stddev)
logp = dist.log_prob(action)
return logp
Training the logp network trains the actor model as well. In the training methods
that are discussed in this section, only the logp network is trained.
As shown in Figure 10.6.7, the entropy model also shares parameters with the
policy network. The output Lambda(1) layer computes the entropy of the Gaussian
distribution given the mean and standard deviation using the following function:
# given the mean and stddev compute the Gaussian dist entropy
def entropy(self, args):
[ 326 ]