Advanced Deep Learning with Keras

Autoencoders
To put the autoencoder in context, x can be an MNIST digit which has a dimension of
28 × 28 × 1 = 784. The encoder transforms the input into a low-dimensional z that can
be a 16-dimension latent vector. The decoder will attempt to recover the input in the
form of x̃ from z. Visually, every MNIST digit x will appear similar to x̃ . Figure 3.1.2
demonstrates this autoencoding process to us. We can observe that the decoded digit
7, while not exactly the same remains close enough.
Since both encoder and decoder are non-linear functions, we can use neural
networks to implement both. For example, in the MNIST dataset, the autoencoder
can be implemented by MLP or CNN. The autoencoder can be trained by minimizing
the loss function through backpropagation. Similar to other neural networks, the
only requirement is that the loss function must be differentiable.
If we treat the input as a distribution, we can interpret the encoder as an encoder of
distribution, p ( z | x ) and the decoder, as the decoder of distribution, p ( x | z ) . The loss
function of the autoencoder is expressed as follows:
( )
L = −log p x | z (Equation 3.1.2)
The loss function simply means that we would like to maximize the chances of
recovering the input distribution given the latent vector distribution. If the decoder
output distribution is assumed to be Gaussian, then the loss function boils down
to MSE since:
m m m
2 2
( z) ∏ ( ̃ ) ( ) ( ) 2
i i
σ ∑ ̃
i i
σ α∑
̃
i i
L = − log p | = − log N x ; x , = − log N x ; x , x − x
x (Equation 3.1.3)
i=
1
2
In this example, ( xi
; x̃
i,
σ )
i= 1 i=
1
N represents a Gaussian distribution with a mean of x̃
i and
2
variance of σ . A constant variance is assumed. The decoder output x̃ is assumed to
i
be independent. While m is the output dimension.
Building autoencoders using Keras
We're now going to move onto something really exciting, building an autoencoder
using Keras library. For simplicity, we'll be using the MNIST dataset for the first set
of examples. The autoencoder will then generate a latent vector from the input data
and recover the input using the decoder. The latent vector in this first example is
16-dim.
[ 74 ]