Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

course) up to a given number of epochs (the aptly named patience argument).
After the scheduler runs out of patience, it updates the learning rate, multiplying it
by the factor argument (for the schedulers listed in the last section, this factor was
named gamma).
To illustrate its behavior, let’s assume the validation loss remains at the same value
(whatever that is) for 12 epochs in a row. What would our scheduler do?
dummy_optimizer = optim.SGD([nn.Parameter(torch.randn(1))], lr=0.01)
dummy_scheduler = ReduceLROnPlateau(
dummy_optimizer, patience=4, factor=0.1
)
Figure 6.25 - Evolution of learning rate (validation loss scheduler)
Its patience is four epochs, so after four epochs observing the same loss, it is
hanging by a thread. Then comes the fifth epoch with no change: "That’s it, the
learning rate must go down," you can almost hear it saying :-) So, in the sixth epoch,
the optimizer is already using the newly updated learning rate. If nothing changes
for four more epochs, it goes down again, as shown in the figure above.
Learning Rates | 481