Mind, Body, World- Foundations of Cognitive Science, 2013a

hidden unit computes its error by summing together all of the error signals that it
receives from the output units to which it is connected. Fifth, once the hidden unit
error has been computed, the weights of the hidden units can be modified using the
same equation that was used to alter the weights of each of the output units.
This procedure can be repeated iteratively if there is more than one layer of
hidden units. That is, the error of each hidden unit in one layer can be propagated
backwards to an adjacent layer as an error signal once the hidden unit weights have
been modified. Learning about this pattern stops once all of the connections have
been modified. Then the next training pattern can be presented to the input units,
and the learning process occurs again.
There are a variety of different ways in which the generic algorithm given above
can be realized. For instance, in stochastic training, connection weights are updated
after each pattern is presented (Dawson, 2004). This approach is called stochastic
because each pattern is presented once per epoch of training, but the order of
presentation is randomized for each epoch. Another approach, batch training, is to
accumulate error over an epoch and to only update weights once at the end of the
epoch, using accumulated error (Rumelhart, Hinton, & Williams, 1986a). As well,
variations of the algorithm exist for different continuous activation functions. For
instance, an elaborated error term is required to train units that have Gaussian activation
functions, but when this is done, the underlying mathematics are essentially
the same as in the original generalized delta rule (Dawson & Schopflocher, 1992b).
New Connectionism was born when the generalized delta rule was invented.
Interestingly, the precise date of its birth and the names of its parents are not completely
established. The algorithm was independently discovered more than once.
Rumelhart, Hinton, and Williams (1986a, 1986b) are its most famous discoverers
and popularizers. It was also discovered by David Parker in 1985 and by Yann LeCun
in 1986 (Anderson, 1995). More than a decade earlier, the algorithm was reported
in Paul Werbos’ (1974) doctoral thesis. The mathematical foundations of the generalized
delta rule can be traced to an earlier decade, in a publication by Shun-Ichi
Amari (1967).
In an interview (Anderson & Rosenfeld, 1998), neural network pioneer Stephen
Grossberg stated that “Paul Werbos, David Parker, and Shun-Ichi Amari should
have gotten credit for the backpropagation model, instead of Rumelhart, Hinton,
and Williams” (pp. 179–180). Regardless of the credit assignment problem associated
with the scientific history of this algorithm, it transformed cognitive science in
the mid-1980s, demonstrating “how the lowly concepts of feedback and derivatives
are the essential building blocks needed to understand and replicate higher-order
phenomena like learning, emotion and intelligence at all levels of the human mind”
(Werbos, 1994, p. 1).
Elements of Connectionist Cognitive Science 161