pdfcoffee

Both the functions allow small updates if x is negative, which might be useful in
certain conditions.
Chapter 1
Activation functions
Sigmoid, Tanh, ELU, LeakyReLU, and ReLU are generally called activation functions
in neural network jargon. In the gradient descent section, we will see that those
gradual changes typical of sigmoid and ReLU functions are the basic building blocks
to develop a learning algorithm that adapts little by little by progressively reducing
the mistakes made by our nets. An example of using the activation function σσ with
(x 1
, x 2
,..., x m
) input vector, (w 1
, w 2
,..., w m
) weight vector, b bias, and ∑ summation is
given in Figure 11. Note that TensorFlow 2.0 supports many activation functions,
a full list of which is available online:
Figure 11: An example of an activation function applied after a linear function
In short – what are neural networks after all?
In one sentence, machine learning models are a way to compute a function that maps
some inputs to their corresponding outputs. The function is nothing more than a
number of addition and multiplication operations. However, when combined with
a non-linear activation and stacked in multiple layers, these functions can learn
almost anything [8]. You also need a meaningful metric capturing what you want to
optimize (this being the so-called loss function that we will cover later in the book),
enough data to learn from, and sufficient computational power.
[ 13 ]