Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Model Configuration & Training
Let’s create an instance of our model using DistilBERT and train it in the usual way:
Model Configuration
1 torch.manual_seed(41)
2 bert_model = AutoModel.from_pretrained("distilbert-base-uncased")
3 model = BERTClassifier(bert_model, 128, n_outputs=1)
4 loss_fn = nn.BCEWithLogitsLoss()
5 optimizer = optim.Adam(model.parameters(), lr=1e-5)
Model Training
1 sbs_bert = StepByStep(model, loss_fn, optimizer)
2 sbs_bert.set_loaders(train_loader, test_loader)
3 sbs_bert.train(1)
You probably noticed that it takes quite some time to train for a single epoch; but,
then again, there are more than 66 million parameters to update:
sbs_bert.count_parameters()
Output
66461441
Let’s check the accuracy of our model:
StepByStep.loader_apply(test_loader, sbs_bert.correct)
Output
tensor([[424, 440],
[317, 331]])
That’s 96.11% accuracy on the validation set—nice! Of course, our dataset is tiny
and the model is huge, but still!
988 | Chapter 11: Down the Yellow Brick Rabbit Hole