Advanced Deep Learning with Keras

Improved GANs
The intuition behind EMD is that it is a measure of how much mass γ ( x,
y)
should be transported by d = ||x - y|| for the probability distribution p data
in order to match the probability distribution p g
. γ( x,
y)
is a joint distribution in
the space of all possible joint distributions ∏ ( pdata,
pg
). γ ( x,
y)
is also known as
a transport plan to reflect the strategy for transporting masses to match the two
probability distributions. There are many possible transport plans given the two
probability distributions. Roughly speaking, inf indicates a transport plan with
the minimum cost.
For example, Figure 5.1.1 shows us two simple discrete distributions x and y . x
has masses m i
for i = 1, 2, 3 and 4 at locations x i
for i = 1, 2, 3 and 4. Meanwhile y
has masses m i
for i =1 and 2 at locations y i
for i = 1 and 2. To match the distribution
y , the arrows show the minimum transport plan to move each mass x i
by d i
. The
EMD is computed as:
4
∑ xidi
0.2( 0.4) 0.3( 0.5) 0.1( 0.3) 0.4( 0.7)
0.54 (Equation 5.1.4)
EMD = = + + + =
i=
1
In Figure 5.1.1, the EMD can be interpreted as the least amount of work needed to
move the pile of dirt x to fill the hole y . While in this example, the inf can also be
deduced from the figure, in most cases especially in continuous distributions, it is
intractable to exhaust all possible transport plans. We will come back to this problem
later on in this chapter. In the meantime, we'll show how the GAN loss functions are,
in fact, minimizing the Jensen-Shannon (JS) divergence.
Distance function in GANs
We're now going to compute the optimal discriminator given any generator from
the loss function in the previous chapter. We'll recall the following equation:
( D)
( )
( x) E ( ( z)
)
L =−E
x~ p
log D −
z
log 1−D G (Equation 4.1.1)
data
Instead of sampling from the noise distribution, the preceding equation can also
be expressed as sampling from the generator distribution:
( D)
( x) E ( ( x)
)
L =−E
log D − log 1−D (Equation 5.1.5)
x~ pdata
x~
pg
To find the minimum
( ) D
L :
[ 128 ]