pdfcoffee

Chapter 8
Just as in a traditional neural network, where the learned parameters are stored as
weight matrices, the RNN's parameters are defined by the three weight matrices
U, V, and W, corresponding to the weights of the input, output, and hidden
states respectively:
Figure 1: (a) Schematic of an RNN cell; (b) RNN cell unrolled
Figure 1(b) shows the same RNN in an "unrolled view". Unrolling just means that we
draw the network out for the complete sequence. The network shown here has three
time steps, suitable for processing three element sequences. Note that the weight
matrices U, V, and W, that we spoke about earlier, are shared between each of the
time steps. This is because we are applying the same operation to different inputs
at each time step. Being able to share these weights across all the time steps greatly
reduces the number of parameters that the RNN needs to learn.
We can also describe the RNN as a computation graph in terms of equations. The
internal state of the RNN at a time t is given by the value of the hidden vector h(t),
which is the sum of the weight matrix W and the hidden state h t-1
at time t-1, and
the product of the weight matrix U and the input x t
at time t, passed through a
tanh activation function. The choice of tanh over other activation functions such
as sigmoid has to do with it being more efficient for learning in practice, and helps
combat the vanishing gradient problem, which we will learn about later in the
chapter.
[ 281 ]