Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"Is it a bit off again?"
Actually, no—since it is standardizing unseen data using statistics computed on
training data, the results above are expected. The mean will be around zero and
the standard deviation will be around one.
Momentum
There is an alternative way of computing running statistics: Instead of using a
simple average, it uses an exponentially weighted moving average (EWMA) of the
statistics.
The naming convention, though, is very unfortunate: The alpha parameter of the
EWMA was named momentum, adding to the confusion. There is even a note in
PyTorch’s documentation warning about this:
"This momentum argument is different from one used in optimizer classes and the
conventional notion of momentum." [128]
The bottom line is: Ignore the confusing naming convention and
think of the "momentum" argument as the alpha parameter of a
regular EWMA.
The documentation also uses x to refer to a particular statistic when introducing
the mathematical formula of "momentum," which does not help at all.
So, to make it abundantly clear what is being computed, I present the formulas
below:
Equation 7.7 - Running statistic
Let’s try it out in practice:
batch_normalizer_mom = nn.BatchNorm1d(
num_features=2, affine=False, momentum=0.1
)
batch_normalizer_mom.state_dict()
542 | Chapter 7: Transfer Learning