Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

This means we could have used a higher learning rate, like 0.005, to train our
model. But this also means we need to recreate the optimizer and update it in
sbs_new. First, let’s create a method for setting its optimizer:
StepByStep Method
def set_optimizer(self, optimizer):
self.optimizer = optimizer
setattr(StepByStep, 'set_optimizer', set_optimizer)
Then, we create and set the new optimizer and train the model as usual:
Updating LR and Model Training
1 new_optimizer = optim.Adam(new_model.parameters(), lr=0.005)
2 sbs_new.set_optimizer(new_optimizer)
3 sbs_new.set_loaders(train_loader, val_loader)
4 sbs_new.train(10)
If you try it out, you’ll find that the training loss actually goes down a bit faster (and
that the model might be overfitting).
DISCLAIMER: The learning rate finder is surely not magic!
Sometimes you’ll not get the "U"-shaped curve: Maybe the initial
learning rate (as defined in your optimizer) is too high already, or
maybe the end_lr is too low. Even if you do, it does not
necessarily mean the mid-point of the descending part will give
you the fastest learning rate for your model.
"OK, if I manage to choose a good learning rate from the start, am I
done with it?"
Sorry, but NO! Well, it depends; it might be fine for simpler (but real, not toy)
problems. The issue here is, for larger models, the loss surface (remember that,
from Chapter 0?) becomes very messy, and a learning rate that works well at the
start of model training may be too high for a later stage of model training. It means
that the learning rate needs to change or adapt.
452 | Chapter 6: Rock, Paper, Scissors