Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

4. Positional Encoding
We haven’t changed the positional encoding module; it is here for the sake of
completion only:
Positional Encoding
1 class PositionalEncoding(nn.Module):
2 def __init__(self, max_len, d_model):
3 super().__init__()
4 self.d_model = d_model
5 pe = torch.zeros(max_len, d_model)
6 position = torch.arange(0, max_len).float().unsqueeze(1)
7 angular_speed = torch.exp(
8 torch.arange(0, d_model, 2).float() *
9 (-np.log(10000.0) / d_model)
10 )
11 # even dimensions
12 pe[:, 0::2] = torch.sin(position * angular_speed)
13 # odd dimensions
14 pe[:, 1::2] = torch.cos(position * angular_speed)
15 self.register_buffer('pe', pe.unsqueeze(0))
16
17 def forward(self, x):
18 # x is N, L, D
19 # pe is 1, maxlen, D
20 scaled_x = x * np.sqrt(self.d_model)
21 encoded = scaled_x + self.pe[:, :x.size(1), :]
22 return encoded
868 | Chapter 10: Transform and Roll Out