Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

The figure below should look familiar: It is a typical recurrent neural network that
we’re using to encode the source sequence.
Figure 9.4 - Encoder
The encoder model is a slim version of our models from Chapter 8: It simply
returns a sequence of hidden states.
Encoder
1 class Encoder(nn.Module):
2 def __init__(self, n_features, hidden_dim):
3 super().__init__()
4 self.hidden_dim = hidden_dim
5 self.n_features = n_features
6 self.hidden = None
7 self.basic_rnn = nn.GRU(self.n_features,
8 self.hidden_dim,
9 batch_first=True)
10
11 def forward(self, X):
12 rnn_out, self.hidden = self.basic_rnn(X)
13
14 return rnn_out # N, L, F
"Don’t we need only the final hidden state?"
That’s correct. We’ll be using the final hidden state only … for now.
In the "Attention" section, we’ll be using all hidden states, and
that’s why we’re implementing the encoder like this.
Let’s go over a simple example of encoding: We start with a sequence of
690 | Chapter 9 — Part I: Sequence-to-Sequence