pdfcoffee

Chapter 8
Notice that the only difference from the equations for the LSTM is the additional c t-1
term for computing outputs of the input (i), forget (f), and output (o) gates:
ii = σσ(WW ii h tt−1 + UU ii xx tt + VV ii cc tt−1 )
ff = σσ(WW ff h tt−1 + UU ff xx tt + VV ff cc tt−1 )
oo = σσ(WW oo h tt−1 + UU oo xx tt + VV oo cc tt−1 )
gg = tanh⁡(WW gg h tt−1 + UU gg xx tt )
cc tt = (ff ∗ cc tt−1 ) + (gg ∗ ii)
h tt = tanh(cc tt ) ∗ oo
TensorFlow 2.0 provides an experimental implementation of the peephole LSTM
cell. To use this in your own RNN layers, you will need to wrap the cell (or list of
cells) in the RNN wrapper, as shown in the following code snippet:
hidden_dim = 256
peephole_cell = tf.keras.experimental.PeepholeLSTMCell(hidden_dim)
rnn_layer = tf.keras.layers.RNN(peephole_cell)
In the previous section, we have seen some RNN cell variants that were developed
to target specific inadequacies of the basic RNN cell. In the next section, we will
look at variations in the architecture of the RNN network itself, which were built to
address specific use cases.
RNN variants
In this section, we will look at a couple of variations on the basic RNN architecture
that can provide performance improvements in some specific circumstances. Note
that these strategies can be applied for different kinds of RNN cells, as well as for
different RNN topologies, which we will learn about later.
Bidirectional RNNs
We have seen how, at any given time step t, the output of the RNN is dependent
on the outputs at all previous time steps. However, it is entirely possible that the
output is also dependent on the future outputs as well. This is especially true for
applications such as natural language processing where the attributes of the word or
phrase we are trying to predict may be dependent on the context given by the entire
enclosing sentence, not just the words that came before it.
[ 289 ]