Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Encoder + Self-Attention
1 class EncoderSelfAttn(nn.Module):
2 def __init__(self, n_heads, d_model,
3 ff_units, n_features=None):
4 super().__init__()
5 self.n_heads = n_heads
6 self.d_model = d_model
7 self.ff_units = ff_units
8 self.n_features = n_features
9 self.self_attn_heads = \
10 MultiHeadAttention(n_heads,
11 d_model,
12 input_dim=n_features)
13 self.ffn = nn.Sequential(
14 nn.Linear(d_model, ff_units),
15 nn.ReLU(),
16 nn.Linear(ff_units, d_model),
17 )
18
19 def forward(self, query, mask=None):
20 self.self_attn_heads.init_keys(query)
21 att = self.self_attn_heads(query, mask)
22 out = self.ffn(att)
23 return out
Remember that the "query" in the forward() method actually gets the data points
from the source sequence. These data points will be transformed into different
"keys," "values," and "queries" inside each of the attention heads. The output of
the attention heads is a context vector (att) that goes through a feed-forward
network to produce a "hidden state."
By the way, now that we’ve gotten rid of the recurrent layer, we’ll
be talking about model dimensions (d_model) instead of hidden
dimensions (hidden_dim). You still get to choose it, though.
The mask argument should receive the source mask; that is, the mask we use to
ignore padded data points in our source sequence.
Self-Attention | 745