Advanced Deep Learning with Keras

Chapter 5
Divergence
Kullback-
Leibler (KL)
5.1.1
Jensen-
Shannon (JS)
5.1.2
Earth-Mover
Distance
(EMD) or
Wasserstein 1
5.1.3
Expression
p ( x)
D ( || ) ~
log data
KL
pdata pg = Ex pdata
p ( x)
p
≠ DKL ( pg || pdata)
= Ex~
p
log
g
p
data
g
g
( x)
( x)
1 pdata
( x)
1
pg
( x)
D ( p p ) = E
~
log
+ E
~
log
= D p p
g
2 pdata ( x) + pg ( x)
2 pdata ( x) + pg
( x)
2 2
( )
JS data g x pdata x p JS g data
( )
W p , p = inf E ⎡
data g ( , )
x−
y ⎤
~
( pdata
, pg
)
x y γ
γ∈∏
⎣ ⎦
∏ is the set of all joint distributions y(x,y) whose
marginal are p data
and p g
.
where ( pdata,
pg
)
Table 5.1.1: The divergence functions between two probability distribution functions p data
and p g
Figure 5.1.1: The EMD is the weighted amount of mass from x to be transported
in order to match the target distribution, y
[ 127 ]