Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

to initialize the hidden state and set the corresponding attribute—the
encoder’s output is batch-first though, and the hidden state must always be
sequence-first, so we permute its first two dimensions.
• The hidden state attribute is used both as an input and as an output of the
recurrent layer.
• The shape of the output must match the shape of the input, namely, a
sequence of length one.
• The forward() method will be called multiple times as we loop over the
generation of the target sequence.
The whole thing is better understood with a hands-on example in code, so it’s time
to try some decoding to generate a target sequence:
torch.manual_seed(21)
decoder = Decoder(n_features=2, hidden_dim=2)
# Initial hidden state will be encoder's final hidden state
decoder.init_hidden(hidden_seq)
# Initial data point is the last element of source sequence
inputs = source_seq[:, -1:]
target_len = 2
for i in range(target_len):
print(f'Hidden: {decoder.hidden}')
out = decoder(inputs) # Predicts coordinates
print(f'Output: {out}\n')
# Predicted coordinates are next step's inputs
inputs = out
Output
Hidden: tensor([[[ 0.3105, -0.5263]]], grad_fn=<SliceBackward>)
Output: tensor([[[-0.2339, 0.4702]]], grad_fn=<ViewBackward>)
Hidden: tensor([[[ 0.3913, -0.6853]]], grad_fn=<StackBackward>)
Output: tensor([[[-0.0226, 0.4628]]], grad_fn=<ViewBackward>)
We created a loop to generate a target sequence of length two, using the
predictions of one step as inputs to the next. The hidden state, however, was
Encoder-Decoder Architecture | 695