Mind, Body, World- Foundations of Cognitive Science, 2013a

The Turing equivalence of connectionist networks has long been established.
McCulloch and Pitts (1943) proved that a network of McCulloch-Pitts neurons
could be used to build the machine head of a universal Turing machine; universal
power was then achieved by providing this system with an external memory. “To
psychology, however defined, specification of the net would contribute all that could
be achieved in that field” (p. 131). More modern results have used the analog nature
of modern processors to internalize the memory, indicating that an artificial neural
network can simulate the entire Turing machine (Siegelmann, 1999; Siegelmann &
Sontag, 1991, 1995).
Modern associationist psychologists have been concerned about the implications
of the terminal meta-postulate and have argued against it in an attempt
to free their theories from its computational shackles (Anderson & Bower, 1973;
Paivio, 1986). The hidden units of modern artificial neural networks break these
shackles by capturing higher-order associations—associations between associations—that
are not defined in a vocabulary restricted to input and output activities.
The presence of hidden units provides enough power to modern networks to firmly
plant them in the class “universal machine” and to make them viable alternatives to
classical simulations.
4.7 What Do Output Unit Activities Represent?
When McCulloch and Pitts (1943) formalized the information processing of neurons,
they did so by exploiting the all-or-none law. As a result, whether a neuron
responded could be interpreted as assigning a “true” or “false” value to some proposition
computed over the neuron’s outputs. McCulloch and Pitts were able to design
artificial neurons capable of acting as 14 of the 16 possible primitive functions on
the two-valued logic that was described in Chapter 2.
McCulloch and Pitts (1943) formalized the all-or-none law by using the Heaviside
step equation as the activation function for their artificial neurons. Modern activation
functions such as the logistic equation provide a continuous approximation of
the step function. It is also quite common to interpret the logistic function in digital,
step function terms. This is done by interpreting a modern unit as being “on” or “off ”
if its activity is sufficiently extreme. For instance, in simulations conducted with my
laboratory software (Dawson, 2005) it is typical to view a unit as being “on” if its activity
is 0.9 or higher, or “off” if its activity is 0.1 or lower.
Digital activation functions, or digital interpretations of continuous activation
functions, mean that pattern recognition is a primary task for artificial neural networks
(Pao, 1989; Ripley, 1996). When a network performs pattern recognition, it
is trained to generate a digital or binary response to an input pattern, where this
152 Chapter 4