Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

1 Sets attention module and adjusts input dimensions of the regression layer
2 Sets the "keys" (and "values") for the attention module
3 Feeds the "query" to the attention mechanism and concatenates it to the
context vector
Let’s go over a simple example in code, using the updated decoder and attention
classes:
full_seq = (torch.tensor([[-1, -1], [-1, 1], [1, 1], [1, -1]])
.float()
.view(1, 4, 2))
source_seq = full_seq[:, :2]
target_seq = full_seq[:, 2:]
torch.manual_seed(21)
encoder = Encoder(n_features=2, hidden_dim=2)
decoder_attn = DecoderAttn(n_features=2, hidden_dim=2)
# Generates hidden states (keys and values)
hidden_seq = encoder(source_seq)
decoder_attn.init_hidden(hidden_seq)
# Target sequence generation
inputs = source_seq[:, -1:]
target_len = 2
for i in range(target_len):
out = decoder_attn(inputs)
print(f'Output: {out}')
inputs = out
Output
Output: tensor([[[-0.3555, -0.1220]]], grad_fn=<ViewBackward>)
Output: tensor([[[-0.2641, -0.2521]]], grad_fn=<ViewBackward>)
The code above does the bare minimum to generate a target sequence using the
attention mechanism. To actually train a model using teacher forcing, we need to
put the two (or three) classes together…
Attention | 729