Mind, Body, World- Foundations of Cognitive Science, 2013a

esponses to determine the trigger features that the hidden units detect. It is also
shown that changing the activation function of a hidden unit can lead to interesting
complexities in defining the notion of a trigger feature, because some kinds of
hidden units capture families of trigger features that require further analysis.
In Section 4.12 we describe how interpreting the internal structure of a network
begins to shed light on the relationship between algorithms and architectures. Also
described is a network that, as a result of training, translates a classical model of a
task into a connectionist one. This illustrates an intertheoretic reduction between
classical and connectionist theories, raising the possibility that both types of theories
can be described in the same architecture.
4.9 Empiricism and Internal Representations
The ability of hidden units to increase the computational power of artificial neural
networks was well known to Old Connectionism (McCulloch & Pitts, 1943). Its problem
was that while a learning rule could be used to train networks with no hidden
units (Rosenblatt, 1958, 1962), no such rule existed for multilayered networks. The
reason that a learning rule did not exist for multilayered networks was because
learning was defined in terms of minimizing the error of unit responses. While it
was straightforward to define output unit error, no parallel definition existed for
hidden unit error. A hidden unit’s error could not be defined because it was not
related to any directly observable outcome (e.g., external behaviour). If a hidden
unit’s error could not be defined, then Old Connectionist rules could not be used to
modify its connections.
The need to define and compute hidden unit error is an example of the credit
assignment problem:
In playing a complex game such as chess or checkers, or in writing a computer
program, one has a definite success criterion—the game is won or lost. But in the
course of play, each ultimate success (or failure) is associated with a vast number of
internal decisions. If the run is successful, how can we assign credit for the success
among the multitude of decisions? (Minsky, 1963, p. 432)
The credit assignment problem that faced Old Connectionism was the inability
to assign the appropriate credit—or more to the point, the appropriate blame—
to each hidden unit for its contribution to output unit error. Failure to solve this
problem prevented Old Connectionism from discovering methods to make their
most powerful networks belong to the domain of empiricism and led to its demise
(Papert, 1988).
The rebirth of connectionist cognitive science in the 1980s (McClelland &
Rumelhart, 1986; Rumelhart & McClelland, 1986c) was caused by the discovery
Elements of Connectionist Cognitive Science 159