Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

• A final linear layer to map the decoder’s outputs back to the original feature
space (the coordinates we’re trying to predict).
We also need to make some small modifications to the encode() and decode()
methods to account for the components above:
Transformer Encoder-Decoder
1 class EncoderDecoderTransf(EncoderDecoderSelfAttn):
2 def __init__(self, encoder, decoder,
3 input_len, target_len, n_features):
4 super(EncoderDecoderTransf, self).__init__(
5 encoder, decoder, input_len, target_len
6 )
7 self.n_features = n_features
8 self.proj = nn.Linear(n_features, encoder.d_model) 1
9 self.linear = nn.Linear(encoder.d_model, n_features) 2
10
11 def encode(self, source_seq, source_mask=None):
12 # Projection
13 source_proj = self.proj(source_seq) 1
14 encoder_states = self.encoder(source_proj, source_mask)
15 self.decoder.init_keys(encoder_states)
16
17 def decode(self, shifted_target_seq,
18 source_mask=None, target_mask=None):
19 # Projection
20 target_proj = self.proj(shifted_target_seq) 1
21 outputs = self.decoder(target_proj,
22 source_mask=source_mask,
23 target_mask=target_mask)
24 # Linear
25 outputs = self.linear(outputs) 2
26 return outputs
1 Projecting features to model dimensionality
2 Final linear transformation from model to feature space
Let’s briefly review the model’s methods:
• encode(): It takes the source sequence and mask and encodes its projection
into a sequence of states that is immediately used to initialize the "keys" (and
832 | Chapter 10: Transform and Roll Out