Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

As expected, the EWMA without correction (red dashed line) is way off at the
beginning, while the regular moving average (black dashed line) tracks the actual
values much closer. The corrected EWMA, though, does a very good job tracking
the actual values from the very beginning. Sure enough, after 19 days, the two
EWMAs are barely distinguishable.
EWMA Meets Gradients
Who cares about temperatures, anyway? Let’s apply the EWMAs to our gradients,
Adam-style!
For each parameter, we compute two EWMAs: one for its gradients, the other for
the square of its gradients. Next, we use both values to compute the adapted
gradient for that parameter:
Equation 6.9 - Adapted gradient
There they are: Adam’s beta1 and beta2 parameters! Its default values, 0.9 and
0.999, correspond to averages of 19 and 1999 periods, respectively.
So, it is a short-term average for smoothing the gradients, and a very long-term
average for scaling the gradients. The epsilon value in the denominator (usually
1e-8) is there only to prevent numerical issues.
Once the adapted gradient is computed, it replaces the actual gradient in the
parameter update:
Equation 6.10 - Parameter update
Clearly, the learning rate (the Greek letter eta) is left untouched!
Moreover, as a result of the scaling, the adapted gradient is likely to be inside the [-
3, 3] range most of the time (this is akin to the standardization procedure but
without subtracting the mean).
Learning Rates | 461