Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"Wait a minute … I thought this book was about PyTorch!" Yes, it is,
but this serves two purposes: first, to introduce the structure of
our task, which will remain largely the same and, second, to show
you the main pain points so you can fully appreciate how much
PyTorch makes your life easier :-)
For training a model, there is a first initialization step (line numbers refer to
Notebook Cell 1.2 code below):
• Random initialization of parameters / weights (we have only two, b and
w)—lines 3 and 4
• Initialization of hyper-parameters (in our case, only learning rate and number of
epochs)—lines 9 and 11
Make sure to always initialize your random seed to ensure the
reproducibility of your results. As usual, the random seed is 42 [42] ,
the (second) least random [43] of all random seeds one could
possibly choose.
For each epoch, there are four training steps (line numbers refer to Notebook Cell
1.2 code below):
• Compute model’s predictions—this is the forward pass—line 15
• Compute the loss, using predictions and labels and the appropriate loss function
for the task at hand—lines 20 and 22
• Compute the gradients for every parameter—lines 25 and 26
• Update the parameters—lines 30 and 31
For now, we will be using batch gradient descent only, meaning, we’ll use all data
points for each one of the four steps above. It also means that going once through
all of the steps is already one epoch. Then, if we want to train our model over 1,000
epochs, we just need to add a single loop.
In Chapter 2, we’ll introduce mini-batch gradient descent, and
then we’ll have to include a second inner loop.
68 | Chapter 1: A Simple Regression Problem