pdfcoffee

Recurrent Neural Networks
This problem can be solved using a bidirectional LSTM, which are essentially two
RNNs stacked on top of each other, one reading the input from left to right, and the
other reading the input from the right to the left. The output at each time step will be
based on the hidden state of both RNNs. Bidirectional RNNs allow the network to
place equal emphasis on the beginning and end of the sequence, and typically results
in performance improvements.
TensorFlow 2.0 provides support for bidirectional RNNs through a bidirectional
wrapper layer. To make a RNN layer bidirectional, all that is needed is to wrap the
layer with this wrapper layer, shown as follows:
self.lstm = tf.keras.layers.Bidirectional(
tf.keras.layers.LSTM(10, return_sequences=True,
input_shape=(5, 10))
)
Stateful RNNs
RNNs can also be stateful, which means that they can maintain state across batches
during training. That is, the hidden state computed for a batch of training data will
be used as the initial hidden state for the next batch of training data. However, this
needs to be explicitly set, since TensorFlow 2.0 (tf.keras) RNNs are stateless by
default, and resets the state after each batch. Setting an RNN to be stateful means
that it can build state across its training sequence and even maintain that state
when doing predictions.
The benefits of using stateful RNNs are smaller network sizes and/or lower
training times. The disadvantage is that we are now responsible for training the
network with a batch size that reflects the periodicity of the data and resetting
the state after each epoch. In addition, data should not be shuffled while training
the network since the order in which the data is presented is relevant for stateful
networks.
To set a RNN layer as stateful, set the named variable stateful to True. In our
example of a one-to-many topology for learning to generate text, we provide
an example of using a stateful RNN. Here, we train using data consisting of
contiguous text slices, so setting the LSTM to stateful means that the hidden state
generated from the previous text chunk is reused for the current text chunk.
In the next section on RNN topologies, we will look at different ways to set up the
RNN network for different use cases.
[ 290 ]