Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Output
array([0.01 , 0.01258925, 0.01584893, 0.01995262,
0.02511886, 0.03162278, 0.03981072, 0.05011872,
0.06309573, 0.07943282, 0.1 ])
"Cool, but how do I change the learning rate of an optimizer?"
Glad you asked! It turns out, we can assign a scheduler to an optimizer, such that it
updates the learning rate as it goes. We’re going to dive deeper into learning rate
schedulers in a couple of sections. For now, it suffices to know that we can make it
follow a sequence of values like the one above using a scheduler that takes a
custom function. Coincidence? I think not! That’s what we’ll be using lr_fn() for:
dummy_model = CNN2(n_feature=5, p=0.3)
dummy_optimizer = optim.Adam(dummy_model.parameters(), lr=start_lr)
dummy_scheduler = LambdaLR(dummy_optimizer, lr_lambda=lr_fn)
The LambdaLR scheduler takes an optimizer and a custom function as arguments
and modifies the learning rate of that optimizer accordingly. To make it happen,
though, we need to call the scheduler’s step() method, but only after calling the
optimizer’s own step() method:
dummy_optimizer.step()
dummy_scheduler.step()
After one step, the learning rate should have been updated to match the second
value in our array (0.01258925). Let’s double-check it using the scheduler’s
get_last_lr() method:
dummy_scheduler.get_last_lr()[0]
Output
0.012589254117941673
448 | Chapter 6: Rock, Paper, Scissors