Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Not quite a "U" shape, but we still can tell that something in the ballpark of
1e-2 is a good starting point.
Adaptive Learning Rate
That’s what the Adam optimizer is actually doing for us—it starts with the learning
rate provided as an argument, but it adapts the learning rate(s) as it goes, tweaking
it in a different way for each parameter in the model. Or does it?
Truth to be told, Adam does not adapt the learning rate—it really adapts the
gradients. But, since the parameter update is given by the multiplication of both
terms, the learning rate and the gradient, this is a distinction without a difference.
Adam combines the characteristics of two other optimizers: SGD (with momentum)
and RMSProp. Like the former, it uses a moving average of gradients instead of
gradients themselves (that’s the first moment, in statistics jargon); like the latter, it
scales the gradients using a moving average of squared gradients (that’s the
second moment, or uncentered variance, in statistics jargon).
But this is not a simple average. It is a moving average. And it is not any moving
average. It is an exponentially weighted moving average (EWMA).
Before diving into EWMAs, though, we need to briefly go over simple moving
averages.
Moving Average (MA)
To compute the moving average of a given feature x over a certain number of
periods, we just have to average the values observed over that many time steps
(from an initial value observed periods-1 steps ago all the way up to the current
value):
Equation 6.1 - Simple moving average
But, instead of averaging the values themselves, let’s compute the average age of
the values. The current value has an age equals one unit of time while the oldest
454 | Chapter 6: Rock, Paper, Scissors