Advanced Deep Learning with Keras

Improved GANs
In the context of GANs, Equation 5.1.18 can be rewritten by sampling from z-noise
distribution and replacing f w
by the discriminator function, D w
:
( data, g ) max E ⎡
~ ( ) ⎤
x p w
E ⎡
z w ( ( ))
⎤
W p p D x D G z (Equation 5.1.19)
= −
w∈W
data ⎣ ⎦ ⎢ ⎣ ⎥ ⎦
We use the bold letter to highlight the generality to multi-dimensional samples. The
final problem we face is how to find the family of functions w ∈ W . The proposed
solution we're going to go over is that at every gradient update, the weights of the
discriminator, w, are clipped between lower and upper bounds, (for example, -0.0,1
and 0.01):
w ← clip( w, − 0.01,0.01)
(Equation 5.1.20)
The small values of w constrains the discriminator to a compact parameter space
thus ensuring Lipschitz continuity.
We can use Equation 5.1.19 as the basis of our new GAN loss functions. EMD
or Wasserstein 1 is the loss function that the generator aims to minimize, and the
cost function that the discriminator tries to maximize (or minimize -W(p data
,p g
)):
( D)
L =− Ex~ p
Dw ( x) + EzDw
( G ( z)
) 5.1.21 (Equation 5.1.21)
data
( G)
( ( ))
L =−E z
D w
G z (Equation 5.1.22)
In the generator loss function, the first term disappears since it is not directly
optimizing with respect to the real data.
Following table shows the difference between the loss functions of GAN and
( D)
( G)
WGAN. For conciseness, we've simplified the notation for L , and L . These
loss functions are used in training the WGAN as shown in Algorithm 5.1.1. Figure
5.1.3 illustrates that the WGAN model is practically the same as the DCGAN model
except for the fake/true data labels and loss functions:
Network Loss Functions Equation
( )
GAN ( D)
L =−E
x~ p
log D x −Ez
log 1−D G
L
( G)
data
=−E z
log D( G( z)
)
( ) ( ( z)
)
4.1.1
4.1.5
[ 132 ]