pdfcoffee

The Math Behind Deep Learning
In this case the sum will not disappear because the change of weights in the hidden
layer is directly affecting the output. Substituting yy oo = δδ oo (zz oo ) and applying the
chain rule, we have:
∂∂∂∂
= ∑(yy
∂∂ww oo − tt oo ) ∂∂δδ oo (zz oo )
iiii
oo
∂∂ww iiii
= ∑(yy oo − tt oo )δδ′ oo (zz oo ) ∂∂∂∂ oo
∂∂∂∂ iiii
oo
The indirect relation between z o
and the internal weights w ij
(Figure 13) is
mathematically expressed by the expansion:
zz oo = ∑ ww jjjj δδ jj (zz jj ) + bb oo =
jj
∑ ww jjjj δδ jj (∑ ww iiii zz ii + bb ii ) + bb oo
jj
ii
since zz jj = ∑ ww iiii zz ii + bb ii
This suggests applying the chain rule again:
ii
∂∂zz oo
∂∂ww iiii
= (chain rule)
= ∂∂zz oo
∂∂yy jj
∂∂yy jj
∂∂ww iiii
= (substituting z 0
)
= ∂∂yy jjww jjjj
∂∂yy jj
∂∂yy jj
= (deriving)
∂∂ww iiii
= ww jjjj
∂∂yy jj
∂∂ww iiii
= (substituting yy jj = δδ jj (zz jj ) )
= ww jjjj
∂∂δδ jj (zz jj )
∂∂ww iiii
= (chain rule)
= ww jjjj δδ′ jj (zz jj ) ∂∂∂∂ jj
= (substituting zz jj = ∑ yy ii ww iiii + bb ii
∂∂∂∂ iiii
ii
[ 558 ]
)