Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

The attention scores are stored in the alphas attribute of the attention model,
which, in turn, is the decoder’s attn attribute. For each step in the target sequence
generation, the corresponding scores are copied to the alphas attribute of the
EncoderDecoderAttn model (line 22).
IMPORTANT: Pay attention (pun very much intended!) to the
shape of the alphas attribute: (N, L target , L source ). For each one out
of N sequences in a mini-batch, there is a matrix, where each
"query" (Q) coming from the target sequence (a row in this
matrix) has as many attention scores as there are "keys" (K) in
the source sequence (the columns in this matrix).
We’ll visualize these matrices shortly. Moreover, a proper
understanding of how attention scores are organized in the
alphas attribute will make it much easier to understand the next
section: "Self-Attention."
Model Configuration & Training
We just have to replace the original classes for both decoder and model with their
attention counterparts, and we’re good to go:
Model Configuration
1 torch.manual_seed(17)
2 encoder = Encoder(n_features=2, hidden_dim=2)
3 decoder_attn = DecoderAttn(n_features=2, hidden_dim=2)
4 model = EncoderDecoderAttn(encoder, decoder_attn,
5 input_len=2, target_len=2,
6 teacher_forcing_prob=0.5)
7 loss = nn.MSELoss()
8 optimizer = optim.Adam(model.parameters(), lr=0.01)
Model Training
1 sbs_seq_attn = StepByStep(model, loss, optimizer)
2 sbs_seq_attn.set_loaders(train_loader, test_loader)
3 sbs_seq_attn.train(100)
732 | Chapter 9 — Part I: Sequence-to-Sequence