Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Let’s see it in code:
max_len = 10
d_model = 8
position = torch.arange(0, max_len).float().unsqueeze(1)
angular_speed = torch.exp(
torch.arange(0, d_model, 2).float() * (-np.log(10000.0) / d_model)
)
encoding = torch.zeros(max_len, d_model)
encoding[:, 0::2] = torch.sin(angular_speed * position)
encoding[:, 1::2] = torch.cos(angular_speed * position)
As you can see, each position is multiplied by several different angular speeds, and
the resulting coordinates (given by the sine and cosine) compose the actual
encoding. Now, instead of plotting the circles, we can directly plot all sine values
(the even dimensions of the encoding) and all cosine values (the odd dimensions of
the encoding) instead.
Figure 9.48 - Positional encoding as a heatmap
The plots on the bottom show the color-coded encoding, ranging from minus one
(dark blue) to zero (green), all the way to one (yellow). I chose to plot them with the
positions on the horizontal axis so you can more easily associate them with the
corresponding curves on the top. In most blog posts, however, you’ll find the
transposed version; that is, with dimensions on the horizontal axis.
Positional Encoding (PE) | 773