Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

• understanding how the dropout probability generates a distribution of
outputs
• observing the effect of train and eval modes in dropout layers
• visualizing the regularizing effect of dropout layers
• using the learning rate range test to find an interval of learning rate candidates
• computing bias-corrected exponentially weighted moving averages of both
gradients and squared gradients to implement adaptive learning rates like the
Adam optimizer
• capturing gradients using register_hook() on tensors of learnable parameters
• capturing parameters using the previously implemented attach_hooks()
method
• visualizing the path taken by different optimizers for updating parameters
• understanding how momentum is computed and its effect on the parameter
update
• (re)discovering the clever look-ahead trick implemented by Nesterov’s
momentum
• learning about different types of schedulers: epoch, validation loss, and minibatch
• including learning rate schedulers in the training loop
• visualizing the impact of a scheduler on the path taken for updating
parameters
Congratulations! You have just learned about the tools commonly used for training
deep learning models: adaptive learning rates, momentum, and learning rate
schedulers. Far from being an exhaustive lesson on this topic, this chapter has
given you a good understanding of the basic building blocks. You have also learned
how dropout can be used to reduce overfitting and, consequently, improve
generalization.
In the next chapter, we’ll learn about transfer learning to leverage the power of
pre-trained models, and we’ll go over some key components of popular
architectures, like 1x1 convolutions, batch normalization layers, and residual
connections.
[94] https://github.com/dvgodoy/PyTorchStepByStep/blob/master/Chapter06.ipynb
[95] https://colab.research.google.com/github/dvgodoy/PyTorchStepByStep/blob/master/Chapter06.ipynb
496 | Chapter 6: Rock, Paper, Scissors