Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Let’s look at the cross-sections before and after we multiplied feature x by 10.
Figure 0.15 - Comparing cross-sections: before and after
What happened here? The red curve got much steeper (larger gradient), and thus
we must use a lower learning rate to safely descend along it.
More important, the difference in steepness between the red
and the black curves increased.
This is exactly what WE NEED TO AVOID!
Do you remember why?
Because the size of the learning rate is limited by the steepest
curve!
How can we fix it? Well, we ruined it by scaling it 10x larger. Perhaps we can make
it better if we scale it in a different way.
Scaling / Standardizing / Normalizing
Different how? There is this beautiful thing called the StandardScaler, which
transforms a feature in such a way that it ends up with zero mean and unit
standard deviation.
How does it achieve that? First, it computes the mean and the standard deviation of
a given feature (x) using the training set (N points):
Step 4 - Update the Parameters | 51