Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 6.29 - Losses for SGD combining momentum and scheduler
It is definitely harder to tell the difference between curves in the same row, except
for the combination of Nesterov’s momentum and cyclical scheduler, which
produced a smoother reduction in the training loss.
Adaptive vs Cycling
Although adaptive learning rates are considered competitors of cyclical learning rates,
this does not prevent you from combining them and cycling learning rates while
using Adam. While Adam adapts the gradients using its EWMAs, the cycling policy
modifies the learning rate itself, so they can work together indeed.
There is much more to learn about in the topic of learning rates: This section is
meant to be only a short introduction to the topic.
Putting It All Together
In this chapter, we were all over the place: data preparation, model configuration,
and model training—a little bit of everything. Starting with a brand-new dataset,
Rock Paper Scissors, we built a method for standardizing the images (for real this
time) using a temporary data loader. Next, we developed a fancier model that
included dropout layers for regularization. Then, we turned our focus to the
training part, diving deeper into learning rates, optimizers, and schedulers. We
implemented many methods: for finding a learning rate, for capturing gradients
and parameters, and for updating the learning rate using a scheduler.
492 | Chapter 6: Rock, Paper, Scissors