Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

The general idea behind regularization is that, if left unchecked, a model will try to
find the "easy way out" (can you blame it?!) to achieve the target. What does it
mean? It means it may end up relying on a handful of features because these
features were found to be more relevant in the training set. Maybe they are, maybe
they aren’t—it could very well be a statistical fluke, who knows, right?
To make the model more robust, some of the features are randomly denied to it, so
it has to achieve the target in a different way. It makes training harder, but it
should result in better generalization; that is, the model should perform better
when handling unseen data (like the data points in the validation set).
The whole thing looks a lot like the randomization of features used in random
forests to perform the splits. Each tree, or even better, each split has access to a
subset of features only.
"How does this, "feature randomization", work in a deep learning
model?"
To illustrate it, let’s build a sequential model with a single nn.Dropout layer:
dropping_model = nn.Sequential(nn.Dropout(p=0.5))
"Why do I need a model for this? Can’t I use the functional form
F.dropout() instead?"
Yes, a functional dropout would go just fine here, but I wanted to illustrate another
point too, so please bear with me. Let’s also create some neatly spaced points to
make it easier to understand the effect of dropout.
spaced_points = torch.linspace(.1, 1.1, 11)
spaced_points
Output
tensor([0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000,
0.8000, 0.9000, 1.0000, 1.1000])
Next, let’s use these points as inputs of our amazingly simple model:
432 | Chapter 6: Rock, Paper, Scissors