pdfcoffee

Chapter 15
Thinking about backpropagation and
convnets
In this section we want to give an intuition behind backpropagation and convnets.
For the sake of simplicity we will focus on an example of convolution with input X of
size 3 × 3, one single filter W of size 2 × 2 with no padding, stride 1, and no dilation
(see Chapter 5, Advanced Convolutional Neural Networks). The generalization is left as
an exercise.
The standard convolution operation is represented in Figure 15. Simply put, the
convolutional operation is the forward pass:
Figure 15: Forward pass for our convnet toy example
Following the intuition of Figure 15, we can now focus our attention to the backward
pass for the current layer. The key assumption is that we receive a backpropagated
signal ∂∂∂∂
∂∂h iiii
as input, and we need to compute ∂∂∂∂
∂∂ww iiii
and ∂∂∂∂
∂∂xx iiii
. This computation is left as
an exercise but please note that each weight in the filter contributes to each pixel in
the output map or, in other words, any change in a weight of a filter affects all the
output pixels.
Thinking about backpropagation and
RNNs
As you remember from Chapter 8, Recurrent Neural Networks, the basic equation for
an RNN is ss tt = tanh⁡(UU xxxx + WW ssss−1 ), the final prediction is yŷ tt = ssssssssssssss(VVss tt ) at step t,
the correct value is y t
, and the error E is the cross-entropy. Here U, V, W are learning
parameters used for the RNNs' equations. These equations can be visualized as in
Figure 16 where we unroll the recurrency. The core idea is that total error is just the
sum of the errors at each time step.
[ 565 ]