Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

In PyTorch, the decoder is implemented as nn.TransformerDecoder, and its
constructor method expects similar arguments: decoder_layer, num_layers,
and an optional normalization layer to normalize (or not) the outputs.
declayer = nn.TransformerDecoderLayer(
d_model=6, nhead=3, dim_feedforward=20
)
dectransf = nn.TransformerDecoder(
declayer, num_layers=1, norm=nn.LayerNorm
)
PyTorch’s decoder also behaves a bit differently than ours, since it does not
(at the time of writing) implement positional encoding for the inputs, and it
does not normalize the outputs by default.
Before putting the encoder and the decoder together, we still have to make a short
pit-stop and address that teeny-tiny detail…
Layer Normalization
Layer normalization was introduced by Jimmy Lei Ba, Jamie Ryan Kiros, and
Geoffrey E. Hinton in their 2016 paper "Layer Normalization," [151] but it only got
really popular after being used in the hugely successful Transformer architecture.
They say: "…we transpose batch normalization into layer normalization by computing
the mean and variance used for normalization from all of the summed inputs to the
neurons in a layer on a single training case" (the highlight is mine).
Simply put: Layer normalization standardizes individual data
points, not features.
This is completely different than the standardizations we’ve performed so far.
Before, each feature, either in the whole training set (using Scikit-learn’s
StandardScaler way back in Chapter 0), or in a mini-batch (using batch norm in
Chapter 7), was standardized to have zero mean and unit standard deviation. In a
tabular dataset, we standardized the columns.
Layer Normalization | 821