Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Output
tensor([[0.3924, 0.8146]], grad_fn=<TanhBackward>)
That’s the updated hidden state!
Now, let’s take a quick sanity check, feeding the same input to the original RNN
cell:
rnn_cell(X[0:1])
Output
tensor([[0.3924, 0.8146]], grad_fn=<TanhBackward>)
Great, the values match.
We can also visualize this sequence of operations, assuming that every hidden
space "lives" in a feature space delimited by the boundaries given by the hyperbolic
tangent. So, the initial hidden state (0, 0) sits at the center of this feature space,
depicted in the left-most plot in the figure below:
Figure 8.8 - Evolution of the hidden state
The transformed hidden state (the output of linear_hidden()) is depicted in the
second plot: It went through an affine transformation. The point in the center
corresponds to t h . In the third plot, we can see the effect of adding t x (the output of
linear_input()): The whole feature space was translated to the right and up. And
then, in the right most plot, the hyperbolic tangent works its magic and brings the
whole feature space back to the (-1, 1) range. That was the first step in the journey
of a hidden state. We’ll do it once again, using the full sequence, after training a
model.
Recurrent Neural Networks (RNNs) | 599