pdfcoffee

Chapter 15
The derivative can be computed as follows:
σσ ′ (zz) = dd dddd ( 1
1 + ee −zz) = 1
(1 + ee −zz ) −2 dd dddd (ee−zz ) =
ee −zz + 1 − 1
(1 + ee −zz )
1
(1 + ee −zz ) (1 + ee −zz ) =
ee −zz
1
(1 + ee −zz ) = ((1 + ee−zz )
(1 + ee −zz ) − 1
(1 + ee −zz ) ) 1
(1 + ee −zz ) =
1
(1 −
(1 + ee −zz ) ) ( 1
(1 + ee −zz ) )
(1 − σσ(zz))σσ(zz)
Therefore the derivative of σσ(zz) can be computed as a very simple form
σσ ′ (zz) = (1 − σσ(zz))σσ(zz) .
Derivative of tanh
Remember that the arctan function is defined as, tanh(zz) = eezz − ee −zz
ee zz + ee−zz as seen in
Figure 7:
Figure 7: Tanh activation function
If you remember that dd dddd eezz = ee zz and dd dddd ee−zz = −ee −zz , then the derivative is
computed as:
dd
dddd tanh(xx) = (eezz + ee −zz )(ee zz + ee −zz ) − (ee zz − ee −zz )(ee zz − ee −zz )
(ee zz + ee −zz ) 2 =
1 − (eezz − ee −zz ) 2
(ee zz + ee −zz ) 2 = 1 − tttttth2 (zz)
[ 549 ]