Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

decode(), with a single one, encode_decode(), that calls the Transformer itself and
runs its output through the last linear layer to transform it into coordinates. Since
the Transformer expects and outputs sequence-first shapes, there is some backand-forth
permuting as well.
def encode_decode(self, source, target,
source_mask=None, target_mask=None):
# Projections
# PyTorch Transformer expects L, N, F
src = self.preprocess(source).permute(1, 0, 2)
tgt = self.preprocess(target).permute(1, 0, 2)
out = self.transf(src, tgt,
src_key_padding_mask=source_mask,
tgt_mask=target_mask)
# Linear
# Back to N, L, D
out = out.permute(1, 0, 2)
out = self.linear(out) # N, L, F
return out
By the way, we’re keeping the masks to a minimum for the sake of simplicity: Only
src_key_padding_mask and tgt_mask are used.
Moreover, we’re implementing a preprocess() method that takes an input
sequence and
• projects the original features into the model dimensionality;
• adds positional encoding and
• (layer) normalizes the result (remember that PyTorch’s implementation does
not normalize the inputs, so we have to do it ourselves).
The full code looks like this:
Transformer
1 class TransformerModel(nn.Module):
2 def __init__(self, transformer,
3 input_len, target_len, n_features):
The PyTorch Transformer | 841