Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

tokens.
Our model takes an instance of a Transformer encoder, a layer of pre-trained
embeddings (not an nn.EmbeddingBag anymore!), and the desired number of
outputs (logits) corresponding to the number of existing classes.
The forward() method takes a mini-batch of tokenized sentences, pre-processes
them, encodes them (featurizer), and outputs logits (classifier). It really works just
like the Vision Transformer from Chapter 10, but now it takes a sequence of words
(tokens) instead of image patches.
Let’s create an instance of our model and train it in the usual way:
Model Configuration
1 torch.manual_seed(33)
2 # Loads the pre-trained GloVe embeddings into an embedding layer
3 torch_embeddings = nn.Embedding.from_pretrained(
4 extended_embeddings
5 )
6 # Creates a Transformer Encoder
7 layer = EncoderLayer(n_heads=2,
8 d_model=torch_embeddings.embedding_dim,
9 ff_units=128)
10 encoder = EncoderTransf(layer, n_layers=1)
11 # Uses both layers above to build our model
12 model = TransfClassifier(torch_embeddings, encoder, n_outputs=1)
13 loss_fn = nn.BCEWithLogitsLoss()
14 optimizer = optim.Adam(model.parameters(), lr=1e-4)
Model Training
1 sbs_transf = StepByStep(model, loss_fn, optimizer)
2 sbs_transf.set_loaders(train_loader, test_loader)
3 sbs_transf.train(10)
fig = sbs_transf.plot_losses()
944 | Chapter 11: Down the Yellow Brick Rabbit Hole