Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Figure 6.15 - Distribution of weights over lags
See the difference? In a simple moving average every value has the same weight;
that is, they contribute equally to the average. But, in an EWMA, more recent
values have larger weights than older ones.
It may not seem like it, but the two averages above have something in common.
The average age of their values is approximately the same. Cool, right?
So, if the average age of the values in a five-period moving average is three, we
should arrive at (approximately) the same value for the age of the values in the
EWMA above. Let’s understand why this is so. Maybe you haven’t noticed it yet,
but a lag of zero corresponds to an age of one unit of time, a lag of one
corresponds to an age of two units of time, and so on. We can use this information
to compute the average age of the values in an EWMA:
Equation 6.6 - Average age of an EWMA
As the total number of observed values (T) grows, the average age approaches the
inverse of alpha. No, I am not demonstrating this here. Yes, I am showing you a
snippet of code that "proves" it numerically :-)
Learning Rates | 457