Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

alphas = F.softmax(scaled_products, dim=-1)
alphas
Output
tensor([[[0.4254, 0.5746]]], grad_fn=<SoftmaxBackward>)
The computation of the context vectors using scaled dot product is usually
depicted like this:
Figure 9.15 - Scaled dot-product attention
If you prefer to see it in code, it looks like this:
def calc_alphas(ks, q):
dims = q.size(-1)
# N, 1, H x N, H, L -> N, 1, L
products = torch.bmm(q, ks.permute(0, 2, 1))
scaled_products = products / np.sqrt(dims)
alphas = F.softmax(scaled_products, dim=-1)
return alphas
Attention | 721