Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 6.26 - Evolution of learning rate (cyclical scheduler)
By the way, two steps means it would complete a full cycle every four mini-batch
updates—that’s completely unreasonable, and is only used here to illustrate the
behavior.
In practice, a cycle should encompass between two and ten
epochs (according to Leslie N. Smith’s paper), so you need to
figure out how many mini-batches your training set contains
(that’s the length of the data loader) and multiply it by the
desired number of epochs in a cycle to get the total number of
steps in a cycle.
In our example, the train loader has 158 mini-batches, so if we want the learning
rate to cycle over five epochs, the full cycle should have 790 steps, and thus
step_size_up should be half that value (395).
486 | Chapter 6: Rock, Paper, Scissors