Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 7.8 - Before batch normalization
mean1, var1 = batch1[0].mean(axis=0), batch1[0].var(axis=0)
mean1, var1
Output
(tensor([0.8443, 0.8810]), tensor([1.0726, 1.0774]))
These features can surely benefit from some standardization. We’ll use
nn.BatchNorm1d to accomplish it:
batch_normalizer = nn.BatchNorm1d(
num_features=2, affine=False, momentum=None
)
batch_normalizer.state_dict()
Output
OrderedDict([('running_mean', tensor([0., 0.])),
('running_var', tensor([1., 1.])),
('num_batches_tracked', tensor(0))])
The num_features argument should match the dimension of the inputs. To keep
matters simple, we won’t be using the affine transformation (affine=False), or
the momentum (more on that later in this section).
The state_dict() of the batch normalizer tells us the initial values for both running
mean and variance, as well as the number of batches it has already used to compute
the running statistics. Let’s see what happens to them after we normalize our first
mini-batch:
538 | Chapter 7: Transfer Learning