pdfcoffee

Chapter 15
3. Backpropagate the error signals; multiply them with the weights in previous
layers and with the gradients of the associated activation functions.
4. Compute the gradients ∂∂∂∂ for all of the parameters θθ based on the backpropagated
error signal and the feed-forward signals from the inputs.
∂∂∂∂
5. Update the parameters using the computed gradients θθ ← θθ − ηη ∂∂∂∂
∂∂∂∂ .
Note that the above algorithm will work for any choice of differentiable error
function E and for any choice of differentiable activation δδ ll function. The only
requirement is that both must be differentiable.
Limit of backpropagation
Gradient descent with backpropagation is not guaranteed to find the global
minimum of the loss function, but only a local minimum. However, this is not
necessarily a problem observed in practical application.
Cross entropy and its derivative
Gradient descent can be used when cross entropy is adopted as the loss function.
As discussed in Chapter 1, Neural Network Foundations with TensorFlow 2.0, the logistic
loss function is defined as:
EE = LL(cc, pp) = − ∑[cc ii ln(pp ii ) + (1 − cc ii ) ln(1 − pp ii )]
ii
Where c refers to one-hot encoded classes (or labels) whereas p refers to
softmax-applied probabilities. Since cross entropy is applied to softmax-applied
probabilities and to one-hot-encoded classes, we need to take into account the
chain rule for computing the gradient with respect to the final weights, score i
.
Mathematically, we have:
∂∂∂∂
= ∂∂∂∂ ∂∂pp ii
∂∂ssssssssss ii ∂∂pp ii ∂∂ssssssssss ii
Computing each part separately, let's start from ∂∂∂∂
∂∂pp ii
:
∂∂∂∂
∂∂pp ii
= ∂∂(−Σ[cc ii ln(pp ii ) + (1 − cc ii ) ln(ii − pp ii )])
∂∂pp ii
= ∂∂(−cc ii ln(pp ii ) + (1 − cc ii ) ln(ii − pp ii )])
∂∂pp ii
[ 561 ]