Mind, Body, World- Foundations of Cognitive Science, 2013a

In spite of the popularity and success of distributed associative memories as
models of human learning and recall (Hinton & Anderson, 1981), they are extremely
limited in power. When networks learn via the Hebb rule, they produce errors when
they are overtrained, are easily confused by correlated training patterns, and do not
learn from their errors (Dawson, 2004). An error-correcting rule called the delta
rule (Dawson, 2004; Rosenblatt, 1962; Stone, 1986; Widrow & Hoff, 1960) can alleviate
some of these problems, but it does not eliminate them. While association
is a fundamental notion in connectionist models, other notions are required by
modern connectionist cognitive science. One of these additional ideas is nonlinear
processing.
4.3 Nonlinear Transformations
John Stuart Mill modified his father’s theory of associationism (Mill & Mill, 1869;
Mill, 1848) in many ways, including proposing a mental chemistry “in which it is
proper to say that the simple ideas generate, rather than . . . compose, the complex
ones” (Mill, 1848, p. 533). Mill’s mental chemistry is an early example of emergence,
where the properties of a whole (i.e., a complex idea) are more than the sum of the
properties of the parts (i.e., a set of associated simple ideas).
The generation of one class of mental phenomena from another, whenever it can
be made out, is a highly interesting fact in mental chemistry; but it no more supersedes
the necessity of an experimental study of the generated phenomenon than a
knowledge of the properties of oxygen and sulphur enables us to deduce those of
sulphuric acid without specific observation and experiment. (Mill, 1848, p. 534)
Mathematically, emergence results from nonlinearity (Luce, 1999). If a system is
linear, then its whole behaviour is exactly equal to the sum of the behaviours of its
parts. The standard pattern associator that was illustrated in Figure 4-1 is an example
of such a system. Each output unit in the standard pattern associator computes
a net input, which is the sum of all of the individual signals that it receives from the
input units. Output unit activity is exactly equal to net input. In other words, output
activity is exactly equal to the sum of input signals in the standard pattern associator.
In order to increase the power of this type of pattern associator—in order to
facilitate emergence—a nonlinear relationship between input and output must be
introduced.
Neurons demonstrate one powerful type of nonlinear processing. The inputs to
a neuron are weak electrical signals, called graded potentials, which stimulate and
travel through the dendrites of the receiving neuron. If enough of these weak graded
potentials arrive at the neuron’s soma at roughly the same time, then their cumulative
effect disrupts the neuron’s resting electrical state. This results in a massive
Elements of Connectionist Cognitive Science 139