pdfcoffee

The Math Behind Deep Learning
Note that by adjusting the weights we can control the "behavior" of the network
and that a small change in a specific w ij
will be propagated through the network
following its topology (see Figure 5, where the edges in bold are the ones impacted
by the small change in a specific w ij
):
Figure 5: Propagating w ij
changes through the network via the edges in bold
Now that we have reviewed some basic concepts of calculus let's start applying
them to deep learning. The first question is how to optimize activation functions.
Well, I am pretty sure that you are thinking about computing the derivative, so
let's do it!
Activation functions
In Chapter 1, Neural Network Foundations with TensorFlow 2.0, we have seen a few
activation functions including sigmoid, tanh, and ReLU. In the following section
we compute the derivative of these activation functions.
Derivative of the sigmoid
Remember that the sigmoid is defined as σσ(zz) =
1
(see Figure 6):
1 + ee−zz Figure 6: Sigmoid activation function
[ 548 ]