pdfcoffee

Recurrent Neural Networks
In chapter 4, we learned about Convolutional Neural Networks (CNNs), and saw
how they exploit the spatial geometry of their inputs. For example, CNNs for images
apply convolutions to initially small patches of the image, and progress to larger
and larger areas of the image using pooling operations. Convolutions and pooling
operations for images are in two dimensions: the width and height. For audio and
text streams, one-dimensional convolution and pooling operations are applied along
the time dimension, and for video streams, these operations are applied in three
dimensions: along the height, width, and time dimensions.
In this chapter, we will focus on Recurrent Neural Networks (RNNs), a class of
neural networks that are popularly used on text inputs. RNNs are very flexible
and have been used to solve problems such as speech recognition, language
modeling, machine translation, sentiment analysis, and image captioning, to name
a few. RNNs exploit the sequential nature of their input. Sequential inputs could
be text, speech, time series, and anything else where the occurrence of an element
in a sequence is dependent on the elements that came before it. In this chapter,
we will see examples of various RNNs, and learn how to implement them with
TensorFlow 2.0.
We will first look at the internals of a basic RNN cell and how it deals with these
sequential dependencies in the input. We will also learn about some limitations
of the basic RNN cell (implemented as SimpleRNN in Keras) and see how two
popular variants of the SimpleRNN cell – the Long Short-Term Memory (LSTM)
and Gated Recurrent Unit (GRU) – overcome this limitation.
We will then zoom out one level and consider the RNN layer itself, which is just
the RNN cell applied to every time step. An RNN can be thought of as a graph of
RNN cells, where each cell performs the same operation on successive elements of
the sequence. We will describe some simple modifications to improve performance,
such as making the RNN bidirectional and/or stateful.
[ 279 ]