Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure E.3 - The effect of batch normalization
The left-most plot shows us the result of a bad initialization scheme: vanished
gradients. The center plot shows us the result of a proper initialization scheme.
Finally, the right-most plot shows us that batch normalization can indeed
compensate for a bad initialization.
Not all bad gradients vanish, though—some bad gradients explode!
Exploding Gradients
The root of the problem is the same: deeper and deeper models. If the model has a
couple of layers only, one large gradient won’t do any harm. But, if there are many
layers, the gradients may end up growing uncontrollably. That’s the so-called
exploding gradients problem, and it’s fairly easy to spot: Just look for NaN values in
the loss. If that’s the case, it means that the gradients grew so large that they cannot
be properly represented anymore.
"Why does it happen?"
There may be a compounding effect (think of raising a relatively small number (e.g.,
1.5) to a large power (e.g., 20)), especially in recurrent neural networks (the topic
of Chapter 8), since the same weights are used repeatedly along a sequence. But,
there are other reasons as well: The learning rate may be too high, or the target
variable (in a regression problem) may have a large range of values. Let me
illustrate it.
Data Generation & Preparation
Let’s use Scikit-Learn’s make_regression() to generate a dataset of 1,000 points
with ten features each, and a little bit of noise:
Vanishing and Exploding Gradients | 571