Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"But what is a hidden state anyway?"
Excellent question! A hidden state is simply a vector. The size of the vector is up to
you. Really. You need to specify the number of hidden dimensions, which means
specifying the size of the vector that represents a hidden state. Let’s create a twodimensional
hidden state just for kicks:
hidden_state = torch.zeros(2)
hidden_state
Output
tensor([0., 0.])
That’s actually a fine example of an initial hidden state, as we’ll see shortly. But,
before diving deep into the journey of a hidden state through an RNN, let’s take a
look at its top-level representation:
Figure 8.5 - Top-level representation of an RNN
If you’ve ever seen figures representing RNNs, you’ve probably bumped into one of
the two versions depicted above: "unrolled" or not. Let’s start with the unrolled
one: It shows a sequence of two data points being fed to an RNN. We can describe
the flow of information inside an RNN in five easy steps:
1. There is an initial hidden state (h i ) that represents the state of the empty
sequence and is usually initialized with zeros (like the one we created above).
2. An RNN cell takes two inputs: the hidden state representing the state of the
592 | Chapter 8: Sequences