Advanced Deep Learning with Keras

Chapter 1
Layers Optimizer Regularizer Train Accuracy, % Test Accuracy, %
256 SGD Dropout(0.2) 97.26 98.00
256 RMSprop Dropout(0.2) 96.72 97.60
256 Adam Dropout(0.2) 96.79 97.40
512 SGD Dropout(0.2) 97.88 98.30
Table 1.5.1: The different SimpleRNN network configurations and performance measures
In many deep neural networks, other members of the RNN family are more
commonly used. For example, Long Short-Term Memory (LSTM) networks have
been used in both machine translation and question answering problems. LSTM
networks address the problem of long-term dependency or remembering relevant
past information to the present output.
Unlike RNNs or SimpleRNN, the internal structure of the LSTM cell is more
complex. Figure 1.5.4 shows a diagram of LSTM in the context of MNIST digit
classification. LSTM uses not only the present input and past outputs or hidden
states; it introduces a cell state, s t
, that carries information from one cell to the
other. Information flow between cell states is controlled by three gates, f t
, i t
and
q t
. The three gates have the effect of determining which information should be
retained or replaced and the amount of information in the past and current input
that should contribute to the current cell state or output. We will not discuss the
details of the internal structure of the LSTM cell in this book. However, an intuitive
guide to LSTM can be found at: http://colah.github.io/posts/2015-08-
Understanding-LSTMs.
The LSTM() layer can be used as a drop-in replacement to SimpleRNN(). If LSTM
is overkill for the task at hand, a simpler version called Gated Recurrent Unit
(GRU) can be used. GRU simplifies LSTM by combining the cell state and hidden
state together. GRU also reduces the number of gates by one. The GRU() function
can also be used as a drop-in replacement for SimpleRNN().
[ 35 ]