pdfcoffee

Neural Network Foundations with TensorFlow 2.0
We need something different; something smoother. We need a function that
progressively changes from 0 to 1 with no discontinuity. Mathematically, this
means that we need a continuous function that allows us to compute the derivative.
You might remember that in mathematics the derivative is the amount by which a
function changes at a given point. For functions with input given by real numbers,
the derivative is the slope of the tangent line at a point on a graph. Later in this
chapter, we will see why derivatives are important for learning, when we talk about
gradient descent.
Activation function – sigmoid
1
The sigmoid function defined as σσ(xx) =
1 + ee −xx and represented in the following
figure has small output changes in the range (0, 1) when the input varies in the range
(−∞, ∞) . Mathematically the function is continuous. A typical sigmoid function is
represented in Figure 6:
Figure 6: A sigmoid function with output in the range (0,1)
A neuron can use the sigmoid for computing the nonlinear function σσ(zz = wwww + bb) .
Note that if z = wx + b is very large and positive, then ee −zz → 0 so σσ(zz) → 1 , while if
z = wx + b is very large and negative ee −zz → ∞ so σσ(zz) → 0 . In other words, a neuron
with sigmoid activation has a behavior similar to the perceptron, but the changes are
gradual and output values such as 0.5539 or 0.123191 are perfectly legitimate. In this
sense, a sigmoid neuron can answer "maybe."
Activation function – tanh
Another useful activation function is tanh. Defined as tanh(zz) = ee zz − ee −zz
ee zz whose
− ee−zz shape is shown in Figure 7, its outputs range from -1 to 1:
[ 10 ]