Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub
Outputtensor([[[ 0.5475, 0.0875, -1.2350]]])Even though the first "key" (K 0 ) is the smallest in size, it is the most wellalignedto the "query," and, overall, is the "key" with the largest dot product.This means that the decoder would pay the most attention to this particularkey. Makes sense, right?Applying the softmax to these values gives us the following attention scores:scores = F.softmax(prod, dim=-1)scoresOutputtensor([[[0.5557, 0.3508, 0.0935]]])Unsurprisingly, the first key got the largest weight. Let’s use these weights tocompute the context vector:Equation 9.8 - Computing the context vectorv = kcontext = torch.bmm(scores, v)contextOutputtensor([[[ 0.5706, -0.0993]]])Better yet, let’s visualize the context vector.Attention | 719
Since the context vector is a weighted sum of the values (or keys, sincewe’re not applying any affine transformations yet), it is only logical that itslocation is somewhere between the other vectors."Why do we need to scale the dot product?"If we don’t, the distribution of attention scores will get too skewed because thesoftmax function is actually affected by the scale of its inputs:dummy_product = torch.tensor([4.0, 1.0])(F.softmax(dummy_product, dim=-1),F.softmax(100*dummy_product, dim=-1))Output(tensor([0.9526, 0.0474]), tensor([1., 0.]))See? As the scale of the dot products grows larger, the resulting distribution of thesoftmax gets more and more skewed.In our case, there isn’t much difference because our vectors have only twodimensions:720 | Chapter 9 — Part I: Sequence-to-Sequence
- Page 694 and 695: Model Configuration & TrainingWe ca
- Page 696 and 697: size = 5weight = torch.ones(size) *
- Page 698 and 699: torch.manual_seed(17)conv_seq = nn.
- Page 700 and 701: Figure 8.32 - Applying dilated filt
- Page 702 and 703: Model Configuration1 torch.manual_s
- Page 704 and 705: We can actually find an expression
- Page 706 and 707: Data Preparation1 def pack_collate(
- Page 708 and 709: and variable-length sequences.Model
- Page 710 and 711: • generating variable-length sequ
- Page 712 and 713: import copyimport numpy as npimport
- Page 714 and 715: Figure 9.3 - Sequence datasetThe co
- Page 716 and 717: coordinates of a "perfect" square a
- Page 718 and 719: Let’s pretend for a moment that t
- Page 720 and 721: to initialize the hidden state and
- Page 722 and 723: predictions in previous steps have
- Page 724 and 725: the second set of predicted coordin
- Page 726 and 727: Let’s create an instance of the m
- Page 728 and 729: Model Configuration & TrainingThe m
- Page 730 and 731: Sure, we can!AttentionHere is a (no
- Page 732 and 733: based on "the" and "zone," I’ve j
- Page 734 and 735: Figure 9.12 - Matching a query to t
- Page 736 and 737: Outputtensor([[[ 0.0832, -0.0356],[
- Page 738 and 739: utmost importance for the correct i
- Page 740 and 741: Its formula is:Equation 9.3 - Cosin
- Page 742 and 743: second hidden state contributes to
- Page 746 and 747: alphas = F.softmax(scaled_products,
- Page 748 and 749: Outputtensor([[[ 0.2138, -0.3175]]]
- Page 750 and 751: Attention Mechanism1 class Attentio
- Page 752 and 753: "Why would I want to force it to do
- Page 754 and 755: 1 Sets attention module and adjusts
- Page 756 and 757: encdec = EncoderDecoder(encoder, de
- Page 758 and 759: fig = sbs_seq_attn.plot_losses()Fig
- Page 760 and 761: Figure 9.20 - Attention scoresSee?
- Page 762 and 763: Wide vs Narrow AttentionThis mechan
- Page 764 and 765: "What’s so special about it?"Even
- Page 766 and 767: Once again, the affine transformati
- Page 768 and 769: Next, we shift our focus to the sel
- Page 770 and 771: Encoder + Self-Attention1 class Enc
- Page 772 and 773: Figure 9.27 - Encoder with self- an
- Page 774 and 775: The figure below depicts the self-a
- Page 776 and 777: shifted_seq = torch.cat([source_seq
- Page 778 and 779: Equation 9.17 - Decoder’s (masked
- Page 780 and 781: At evaluation / prediction time we
- Page 782 and 783: Outputtensor([[[0.4132, 0.3728],[0.
- Page 784 and 785: Figure 9.33 - Encoder + decoder + a
- Page 786 and 787: 64 return outputsThe encoder-decode
- Page 788 and 789: Figure 9.34 - Losses—encoder + de
- Page 790 and 791: curse. On the one hand, it makes co
- Page 792 and 793: "Are we done now? Is this good enou
Output
tensor([[[ 0.5475, 0.0875, -1.2350]]])
Even though the first "key" (K 0 ) is the smallest in size, it is the most wellaligned
to the "query," and, overall, is the "key" with the largest dot product.
This means that the decoder would pay the most attention to this particular
key. Makes sense, right?
Applying the softmax to these values gives us the following attention scores:
scores = F.softmax(prod, dim=-1)
scores
Output
tensor([[[0.5557, 0.3508, 0.0935]]])
Unsurprisingly, the first key got the largest weight. Let’s use these weights to
compute the context vector:
Equation 9.8 - Computing the context vector
v = k
context = torch.bmm(scores, v)
context
Output
tensor([[[ 0.5706, -0.0993]]])
Better yet, let’s visualize the context vector.
Attention | 719
Change language