Advanced Deep Learning with Keras

Variational Autoencoders (VAEs)
Typically, Qφ ( z | x)
is chosen to be a multivariate Gaussian:
( )
( | ) ; ( ),
( ( ))
Qθ z x = N z µ x diag σ x (Equation 8.1.6)
Both mean, µ ( x)
, and standard deviation, σ ( x)
, are computed by the encoder neural
network using the input data points. The diagonal matrix implies that the elements
of z are independent.
Core equation
The inference model Qφ ( z | x)
generates latent vector z from input x. Qφ ( z | x)
is like
the encoder in an autoencoder model. On the other hand, Pθ ( x | z)
reconstructs the
input from the latent code z. Pθ ( x | z)
acts like the decoder in an autoencoder model.
To estimate Pθ ( x)
, we must identify its relationship with Qφ ( z | x)
and P ( x | z)
If Qφ ( z | x)
is an estimate of Pθ
( z | x)
, the Kullback-Leibler (KL) divergence
determines the distance between these two conditional densities:
( φ ( | ) ||
θ ( | )) = ⎡
z~
Q
log
φ ( | ) −log θ ( | ) ⎤
DKL
Q z x P z x
⎣
Q z x P z x
⎦
Using Bayes theorem,
θ .
E (Equation 8.1.7)
in Equation 8.1.7,
( )
P z x
θ
( ) ( )
P x z P z
θ θ
= (Equation 8.1.8)
( )
P x
θ
DKL
( Qφ ( z | x) || Pθ ( z | x)
) = ⎡
z~
Q
log Q ( z | x) log P ( x | z) log P ( z) ⎤
⎣ φ
−
θ
−
θ ⎦
+ log Pθ
( x)
( )
E (Equation 8.1.9)
log Pθ x can be taken out the expectation since it is not dependent
on z ~ Q . Rearranging the preceding equation and recognizing that
E ⎡
z~ Q
log Q ( z | x) log P ( z) ⎤
⎣ φ
−
θ ⎦
= DKL
( Qφ ( z | x) Pθ
( z)
) :
( ) ( ) ( ( ) ( ))
log Pθ ( x) − DKL Qφ ( z | x) || Pθ ( z | x)
= ⎡
z~
Q
log P x | z ⎤
⎣ θ ⎦
− DKL
Qφ z | x || Pθ
z
E (Equation 8.1.10)
[ 240 ]