Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

if callable(self.clipping): 1
self.clipping()
1
# Step 4 - Updates parameters
self.optimizer.step()
self.optimizer.zero_grad()
# Returns the loss
return loss.item()
# Returns the function that will be called inside the train loop
return perform_train_step_fn
setattr(StepByStep, 'set_clip_grad_value', set_clip_grad_value)
setattr(StepByStep, 'set_clip_grad_norm', set_clip_grad_norm)
setattr(StepByStep, 'remove_clip', remove_clip)
setattr(StepByStep, '_make_train_step_fn', _make_train_step_fn)
1 Gradient clipping after computing gradients and before updating parameters
The gradient clipping methods above work just fine for most models, but they are
of little use for recurrent neural networks (we’ll discuss them in Chapter 8), which
require gradients to be clipped during backpropagation. Fortunately, we can use
the backward hooks code to accomplish that:
Vanishing and Exploding Gradients | 579