Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

We can set the parameter a using the functional version (argument weight in
F.prelu()):
dummy_z = torch.tensor([-3., 0., 3.])
F.prelu(dummy_z, weight=torch.tensor(0.25))
Output
tensor([-0.7500, 0.0000, 3.0000])
But, in the regular module (nn.PReLU), it doesn’t make sense to set it, since it is going
to be learned, right? We can still set the initial value for it, though:
nn.PReLU(init=0.25)(dummy_z)
Output
tensor([-0.7500, 0.0000, 3.0000], grad_fn=<PreluBackward>)
Did you notice the grad_fn attribute on the resulting tensor? It shouldn’t be a
surprise, after all—where there is learning, there is a gradient.
Deep Model
Now that we’ve learned that activation functions break the equivalence to a
shallow model, let’s use them to transform our former deep-ish model into a real
deep model. It has the same architecture as the previous model, except for the
activation functions applied to the outputs of the hidden layers. Here is the
diagram of the updated model.
320 | Chapter 4: Classifying Images