pdfcoffee

The Math Behind Deep Learning
Therefore the derivative of tanh(z) can be computed as a very simple form:
tttttth ′ (zz) = 1 − tttttth 2 (zz) .
Derivative of ReLU
The ReLU function is defined as f(x) = max(0, x) (see Figure 8). The derivative of
ReLU is:
ff ′ (xx) = { 1,
0,
iiiiii > 0
ooooheeeeeeeeeeee
Note that ReLU is non-differentiable at zero. However, it is differentiable anywhere
else, and the value of the derivative at zero can be arbitrarily chosen to be a 0 or 1,
as demonstrated in Figure 8:
Figure 8: ReLU activation function
Backpropagation
Now that we have computed the derivative of the activation functions, we can
describe the backpropagation algorithm – the mathematical core of deep learning.
Sometimes, backpropagation is called backprop for short.
[ 550 ]