pdfcoffee

Recurrent Neural Networks
If you would like to run this code yourself, you can find the code in the code folder
for this chapter. In order to run it from the command line, enter the following
command. The output is written to the console:
$ python gru_pos_tagger.py
Now that we have seen some examples of three common RNN network topologies,
let us explore the most popular of them all – the seq2seq model, also known as the
Recurrent encoder-decoder architecture.
Encoder-Decoder architecture – seq2seq
The example of a many-to-many network we just saw was mostly similar to the
many-to-one network. The one important difference was that the RNN returns
outputs at each time step instead of a single combined output at the end. One other
noticeable feature was that the number of input time steps was equal to the number
of output time steps. As you learn about the encoder-decoder architecture, which is
the "other," and arguably more popular, style of a many-to-many network, you will
notice another difference – the output is in line with the input in a many-to-many
network, that is, it is not necessary for the network to wait until all of the input is
consumed before generating the output.
The Encoder-Decoder architecture is also called a seq2seq model. As the name
implies, the network is composed of an encoder and a decoder part, both RNNbased,
and capable of consuming and returning sequences of outputs corresponding
to multiple time steps. The biggest application of the seq2seq network has been in
neural machine translation, although it is equally applicable for problems that can be
roughly structured as translation problems. Some examples are sentence parsing
[10] and image captioning [24]. The seq2seq model has also been used for time series
analysis [25] and question answering.
In the seq2seq model, the encoder consumes the source sequence, which is a batch
of integer sequences. The length of the sequence is the number of input time steps,
which corresponds to the maximum input sequence length (padded or truncated
as necessary). Thus the dimensions of the input tensor is (batch_size, number_of_
encoder_timesteps). This is passed into an embedding layer, which will convert
the integer at each time step to an embedding vector. The output of the embedding
is a tensor of shape (batch_size, number_of_encoder_timesteps, encoder_
embedding_dim).
[ 316 ]