Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub
• training the Transformer to tackle our sequence-to-sequence problem• understanding that the validation loss may be much lower than the trainingloss due to regularizing effect of dropout• training another model using PyTorch’s (norm-last) Transformer class• using the Vision Transformer architecture to tackle an image classificationproblem• splitting an image into flattened patches by either rearranging or embeddingthem• adding a special classifier token to the embeddings• using the encoder’s output corresponding to the special classifier token asfeatures for the classifierCongratulations! You’ve just assembled and trained your first Transformer (andeven a cutting-edge Vision Transformer!): This is no small feat. Now you know what"layers" and "sub-layers" stand for and how they’re brought together to build aTransformer. Keep in mind, though, that you may find slightly differentimplementations around. It may be either norm-first or norm-last or maybe yetanother customization. The details may be different, but the overall conceptremains: It is all about stacking attention-based "layers.""Hey, what about BERT? Shouldn’t we use Transformers to tackle NLPproblems?"I was actually waiting for this question: Yes, we should, and we will, in the nextchapter. As you have seen, it is already hard enough to understand the Transformereven when it’s used to tackle such a simple sequence-to-sequence problem as ours.Trying to train a model to handle a more complex natural language processingproblem would only make it even harder.In the next chapter, we’ll start with some NLP concepts and techniques like tokens,tokenization, word embeddings, and language models, and work our way up tocontextual word embeddings, GPT-2, and BERT. We’ll be using several Pythonpackages, including the famous HuggingFace :-)[146] https://github.com/dvgodoy/PyTorchStepByStep/blob/master/Chapter10.ipynb[147] https://colab.research.google.com/github/dvgodoy/PyTorchStepByStep/blob/master/Chapter10.ipynb[148] https://arxiv.org/abs/1906.04341[149] https://arxiv.org/abs/1706.03762Recap | 877
[150] http://nlp.seas.harvard.edu/2018/04/03/attention[151] https://arxiv.org/abs/1607.06450[152] https://arxiv.org/abs/2010.11929[153] https://github.com/arogozhnikov/einops[154] https://amaarora.github.io/2021/01/18/ViT.html[155] https://github.com/lucidrains/vit-pytorch878 | Chapter 10: Transform and Roll Out
- Page 852 and 853: Figure 10.10 - Layer norm vs batch
- Page 854 and 855: Outputtensor([[[ 1.4636, 2.3663],[
- Page 856 and 857: The TransformerLet’s start with t
- Page 858 and 859: "values") in the decoder.• decode
- Page 860 and 861: Data Preparation1 # Generating trai
- Page 862 and 863: Figure 10.15 - Losses—Transformer
- Page 864 and 865: • First, and most important, PyTo
- Page 866 and 867: decode(), with a single one, encode
- Page 868 and 869: 46 for i in range(self.target_len):
- Page 870 and 871: Figure 10.18 - Losses - PyTorch’s
- Page 872 and 873: Figure 10.20 - Sample image—label
- Page 874 and 875: 4041 # Builds a weighted random sam
- Page 876 and 877: Figure 10.23 - Sample image—split
- Page 878 and 879: Einops"There is more than one way t
- Page 880 and 881: Figure 10.26 - Two patch embeddings
- Page 882 and 883: Now each sequence has ten elements,
- Page 884 and 885: It takes an instance of a Transform
- Page 886 and 887: Putting It All TogetherIn this chap
- Page 888 and 889: 1. Encoder-DecoderThe encoder-decod
- Page 890 and 891: This is the actual encoder-decoder
- Page 892 and 893: 3. DecoderThe Transformer decoder h
- Page 894 and 895: 5. Encoder "Layer"The encoder "laye
- Page 896 and 897: 7. "Sub-Layer" WrapperThe "sub-laye
- Page 898 and 899: 8. Multi-Headed AttentionThe multi-
- Page 900 and 901: Model Configuration & TrainingModel
- Page 904 and 905: Part IVNatural Language Processing|
- Page 906 and 907: Additional SetupThis is a special c
- Page 908 and 909: "Down the Yellow Brick Rabbit Hole"
- Page 910 and 911: The actual texts of the books are c
- Page 912 and 913: "What is this punkt?"That’s the P
- Page 914 and 915: 14 # If there is a configuration fi
- Page 916 and 917: Sentence Tokenization in spaCyBy th
- Page 918 and 919: AttributesThe Dataset has many attr
- Page 920 and 921: Output{'labels': 1,'sentence': 'The
- Page 922 and 923: elements from the text. But preproc
- Page 924 and 925: Data AugmentationLet’s briefly ad
- Page 926 and 927: The corpora’s dictionary is not a
- Page 928 and 929: Finally, if we want to convert a li
- Page 930 and 931: Once we’re happy with the size an
- Page 932 and 933: from transformers import BertTokeni
- Page 934 and 935: "What about the separation token?"T
- Page 936 and 937: The last output, attention_mask, wo
- Page 938 and 939: Outputtensor([[ 3, 27, 1, ..., 0, 0
- Page 940 and 941: vector, right? And our vocabulary i
- Page 942 and 943: Maybe you filled this blank in with
- Page 944 and 945: Continuous Bag-of-Words (CBoW)In th
- Page 946 and 947: That’s a fairly simple model, rig
- Page 948 and 949: Figure 11.13 - Continuous bag-of-wo
- Page 950 and 951: Figure 11.15 - Reviewing restaurant
• training the Transformer to tackle our sequence-to-sequence problem
• understanding that the validation loss may be much lower than the training
loss due to regularizing effect of dropout
• training another model using PyTorch’s (norm-last) Transformer class
• using the Vision Transformer architecture to tackle an image classification
problem
• splitting an image into flattened patches by either rearranging or embedding
them
• adding a special classifier token to the embeddings
• using the encoder’s output corresponding to the special classifier token as
features for the classifier
Congratulations! You’ve just assembled and trained your first Transformer (and
even a cutting-edge Vision Transformer!): This is no small feat. Now you know what
"layers" and "sub-layers" stand for and how they’re brought together to build a
Transformer. Keep in mind, though, that you may find slightly different
implementations around. It may be either norm-first or norm-last or maybe yet
another customization. The details may be different, but the overall concept
remains: It is all about stacking attention-based "layers."
"Hey, what about BERT? Shouldn’t we use Transformers to tackle NLP
problems?"
I was actually waiting for this question: Yes, we should, and we will, in the next
chapter. As you have seen, it is already hard enough to understand the Transformer
even when it’s used to tackle such a simple sequence-to-sequence problem as ours.
Trying to train a model to handle a more complex natural language processing
problem would only make it even harder.
In the next chapter, we’ll start with some NLP concepts and techniques like tokens,
tokenization, word embeddings, and language models, and work our way up to
contextual word embeddings, GPT-2, and BERT. We’ll be using several Python
packages, including the famous HuggingFace :-)
[146] https://github.com/dvgodoy/PyTorchStepByStep/blob/master/Chapter10.ipynb
[147] https://colab.research.google.com/github/dvgodoy/PyTorchStepByStep/blob/master/Chapter10.ipynb
[148] https://arxiv.org/abs/1906.04341
[149] https://arxiv.org/abs/1706.03762
Recap | 877
Change language