Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Now we can proceed as usual and feed the padded sequences to an RNN and look
at the results:
torch.manual_seed(11)
rnn = nn.RNN(2, 2, batch_first=True)
output_padded, hidden_padded = rnn(padded)
output_padded
Output
tensor([[[-0.6388, 0.8505],
[-0.4215, 0.8979],
[ 0.3792, 0.3432],
[ 0.3161, -0.1675]],
[[ 0.2911, -0.1811],
[ 0.3051, 0.7055],
[ 0.0052, 0.5819],
[-0.0642, 0.6012]],
[[ 0.3385, 0.5927],
[-0.3875, 0.9422],
[-0.4832, 0.6595],
[-0.1007, 0.5349]]], grad_fn=<PermuteBackward>)
Since the sequences were padded to four data points each, we got four hidden
states for each sequence as output.
"How come the hidden states for the padded points are different
from the hidden state of the last real data point?"
Even though each padded point is just a bunch of zeros, it doesn’t mean it won’t
change the hidden state. The hidden state itself gets transformed, and, even if the
padded point is full of zeros, its corresponding transformation may include a bias
term that gets added nonetheless. This isn’t necessarily a problem, but, if you don’t
like padded points modifying your hidden state, you can prevent that by packing
the sequence instead.
656 | Chapter 8: Sequences