pdfcoffee

The Math Behind Deep Learning
Some mathematical tools
Before introducing backpropagation, we need to review some mathematical tools
from calculus. Don't worry too much; we'll briefly review a few areas, all of which
are commonly covered in high school-level mathematics.
Derivatives and gradients everywhere
Derivatives are a powerful mathematical tool. We are going to use derivatives and
gradients for optimizing our network. Let's look at the definition. The derivative
of a function y = f(x) of a variable x is a measure of the rate at which the value y of
the function changes with respect to the change of the variable x. If x and y are real
numbers, and if the graph of f is plotted against x, the derivative is the "slope" of
this graph at each point.
If the function is linear, y = f(x) = ax + b, the slope is aa = ∆yy . This is a simple result of
∆xx
calculus that can be derived by considering that:
yy + ∆(yy) = ff(xx + ∆xx) = aa(xx + ∆xx) + bb = aaaa + aa∆xx + bb = yy + aa∆xx
∆(yy) = aa∆(xx)
aa = ∆yy
∆xx
In Figure 1 we show the geometrical meaning of ∆ xx , ∆ yy and the angle Θ between the
linear function and the x-cartesian axis:
Figure 1: An example of linear function and rate of change
[ 544 ]