Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

all corners, it gets every noisy square right.
"Can we pick one sequence and observe its hidden state from its
initial to its final values?"
Sure we can!
The Journey of a Hidden State
Let’s use the ABCD sequence of the "perfect" square for this. The initial hidden
state is (0, 0) by default, and it is colored black. Every time a new data point (corner)
is going to be used in the computation, the affected hidden state is colored
accordingly (gray, green, blue, and red, in order).
The figure below tracks the progress of the hidden state over every operation
performed inside the RNN.
The first column has the hidden state that’s an input for the RNN cell at a given
step; the second column has the transformed hidden state; the third, the
translated hidden state (by adding the transformed input); and the last, the
activated hidden state.
There are four rows, one for each data point (corner) in our sequence. The initial
hidden state of each row is the activated state of the previous row, so it starts at
the initial hidden state of the whole sequence (0, 0) and, after processing the gray,
green, blue, and red corners, ends at the final hidden state, the red dot close to (-1,
1) in the last plot.
622 | Chapter 8: Sequences