Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub
Additional SetupThis is a special chapter when it comes to its setup: We won’t be using only PyTorchbut rather a handful of other packages as well, including the de facto standard forNLP tasks—HuggingFace.Before proceeding, make sure you have all of them installed by running thecommands below:!pip install gensim==3.8.3!pip install allennlp==0.9.0!pip install flair==0.8.0.post1 # uses PyTorch 1.7.1!pip install torchvision==0.8.2# HuggingFace!pip install transformers==4.5.1!pip install datasets==1.6.0Some packages, like flair, may have strict dependencies andeventually require the downgrading of some other packages inyour environment, even PyTorch itself.The versions above were used to generate the outputs presentedin this chapter, but you can use newer versions if you want(except for the allennlp package since this specific version isrequired by flair for retrieving ELMo embeddings).ImportsFor the sake of organization, all libraries needed throughout the code used in anygiven chapter are imported at its very beginning. For this chapter, we’ll need thefollowing imports:import osimport jsonimport errnoimport requestsimport numpy as npfrom copy import deepcopyfrom operator import itemgetterJupyter Notebook | 881
import torchimport torch.optim as optimimport torch.nn as nnimport torch.nn.functional as Ffrom torch.utils.data import DataLoader, Dataset, random_split, \TensorDatasetfrom data_generation.nlp import ALICE_URL, WIZARD_URL, download_textfrom stepbystep.v4 import StepByStep# These are the classes we built in Chapters 9 and 10from seq2seq import *import nltkfrom nltk.tokenize import sent_tokenizeimport gensimfrom gensim import corpora, downloaderfrom gensim.parsing.preprocessing import *from gensim.utils import simple_preprocessfrom gensim.models import Word2Vecfrom flair.data import Sentencefrom flair.embeddings import ELMoEmbeddings, WordEmbeddings, \TransformerWordEmbeddings, TransformerDocumentEmbeddingsfrom datasets import load_dataset, Splitfrom transformers import (DataCollatorForLanguageModeling,BertModel, BertTokenizer, BertForSequenceClassification,DistilBertModel, DistilBertTokenizer,DistilBertForSequenceClassification,AutoModelForSequenceClassification,AutoModel, AutoTokenizer, AutoModelForCausalLM,Trainer, TrainingArguments, pipeline, TextClassificationPipeline)from transformers.pipelines import SUPPORTED_TASKS882 | Chapter 11: Down the Yellow Brick Rabbit Hole
- Page 856 and 857: The TransformerLet’s start with t
- Page 858 and 859: "values") in the decoder.• decode
- Page 860 and 861: Data Preparation1 # Generating trai
- Page 862 and 863: Figure 10.15 - Losses—Transformer
- Page 864 and 865: • First, and most important, PyTo
- Page 866 and 867: decode(), with a single one, encode
- Page 868 and 869: 46 for i in range(self.target_len):
- Page 870 and 871: Figure 10.18 - Losses - PyTorch’s
- Page 872 and 873: Figure 10.20 - Sample image—label
- Page 874 and 875: 4041 # Builds a weighted random sam
- Page 876 and 877: Figure 10.23 - Sample image—split
- Page 878 and 879: Einops"There is more than one way t
- Page 880 and 881: Figure 10.26 - Two patch embeddings
- Page 882 and 883: Now each sequence has ten elements,
- Page 884 and 885: It takes an instance of a Transform
- Page 886 and 887: Putting It All TogetherIn this chap
- Page 888 and 889: 1. Encoder-DecoderThe encoder-decod
- Page 890 and 891: This is the actual encoder-decoder
- Page 892 and 893: 3. DecoderThe Transformer decoder h
- Page 894 and 895: 5. Encoder "Layer"The encoder "laye
- Page 896 and 897: 7. "Sub-Layer" WrapperThe "sub-laye
- Page 898 and 899: 8. Multi-Headed AttentionThe multi-
- Page 900 and 901: Model Configuration & TrainingModel
- Page 902 and 903: • training the Transformer to tac
- Page 904 and 905: Part IVNatural Language Processing|
- Page 908 and 909: "Down the Yellow Brick Rabbit Hole"
- Page 910 and 911: The actual texts of the books are c
- Page 912 and 913: "What is this punkt?"That’s the P
- Page 914 and 915: 14 # If there is a configuration fi
- Page 916 and 917: Sentence Tokenization in spaCyBy th
- Page 918 and 919: AttributesThe Dataset has many attr
- Page 920 and 921: Output{'labels': 1,'sentence': 'The
- Page 922 and 923: elements from the text. But preproc
- Page 924 and 925: Data AugmentationLet’s briefly ad
- Page 926 and 927: The corpora’s dictionary is not a
- Page 928 and 929: Finally, if we want to convert a li
- Page 930 and 931: Once we’re happy with the size an
- Page 932 and 933: from transformers import BertTokeni
- Page 934 and 935: "What about the separation token?"T
- Page 936 and 937: The last output, attention_mask, wo
- Page 938 and 939: Outputtensor([[ 3, 27, 1, ..., 0, 0
- Page 940 and 941: vector, right? And our vocabulary i
- Page 942 and 943: Maybe you filled this blank in with
- Page 944 and 945: Continuous Bag-of-Words (CBoW)In th
- Page 946 and 947: That’s a fairly simple model, rig
- Page 948 and 949: Figure 11.13 - Continuous bag-of-wo
- Page 950 and 951: Figure 11.15 - Reviewing restaurant
- Page 952 and 953: You got that right—arithmetic—r
- Page 954 and 955: There we go, 50 dimensions! It’s
Additional Setup
This is a special chapter when it comes to its setup: We won’t be using only PyTorch
but rather a handful of other packages as well, including the de facto standard for
NLP tasks—HuggingFace.
Before proceeding, make sure you have all of them installed by running the
commands below:
!pip install gensim==3.8.3
!pip install allennlp==0.9.0
!pip install flair==0.8.0.post1 # uses PyTorch 1.7.1
!pip install torchvision==0.8.2
# HuggingFace
!pip install transformers==4.5.1
!pip install datasets==1.6.0
Some packages, like flair, may have strict dependencies and
eventually require the downgrading of some other packages in
your environment, even PyTorch itself.
The versions above were used to generate the outputs presented
in this chapter, but you can use newer versions if you want
(except for the allennlp package since this specific version is
required by flair for retrieving ELMo embeddings).
Imports
For the sake of organization, all libraries needed throughout the code used in any
given chapter are imported at its very beginning. For this chapter, we’ll need the
following imports:
import os
import json
import errno
import requests
import numpy as np
from copy import deepcopy
from operator import itemgetter
Jupyter Notebook | 881