Mind, Body, World- Foundations of Cognitive Science, 2013a

idea is to give the network as many degrees of freedom as possible to discover representational
regularities that have not been imposed or predicted by the researcher.
These decisions all involve the architectural level of investigation.
One issue, though, is that networks are greedy, in the sense that they will exploit
whatever resources are available to them. As a result, fairly idiosyncratic and specialized
detectors are likely to be found if too many hidden units are provided to the
network, and the network’s performance may not transfer well when presented with
novel stimuli. To deal with this, one must impose constraints by looking for the simplest
network that will reliably learn the mapping of interest. The idea here is that
such a network might be the one most likely to discover a representation general
enough to transfer the network’s ability to new patterns.
Importantly, sometimes when one makes architectural decisions to seek the
simplest network capable of solving a problem, one discovers that the required network
is merely a perceptron that does not employ any hidden units. In the remaining
sections of this chapter I provide some examples of simple networks that are
capable of performing interesting tasks. In section 4.15 the relevance of perceptrons
to modern theories of associative learning is described. In section 4.16 I present a
perceptron model of the reorientation task. In section 4.17 an interpretation is given
for the structure of a perceptron that learns a seemingly complicated progression
of musical chords.
4.15 New Powers of Old Networks
The history of artificial neural networks can be divided into two periods, Old
Connectionism and New Connectionism (Medler, 1998). New Connectionism
studies powerful networks consisting of multiple layers of units, and connections
are trained to perform complex tasks. Old Connectionism studied networks that
belonged to one of two classes. One was powerful multilayer networks that were
hand wired, not trained (McCulloch & Pitts, 1943). The other was less powerful
networks that did not have hidden units but were trained (Rosenblatt, 1958, 1962;
Widrow, 1962; Widrow & Hoff, 1960).
Perceptrons (Rosenblatt, 1958, 1962) belong to Old Connectionism. A perceptron
is a standard pattern associator whose output units employ a nonlinear activation
function. Rosenblatt’s perceptrons used the Heaviside step function to convert
net input into output unit activity. Modern perceptrons use continuous nonlinear
activation functions, such as the logistic or the Gaussian (Dawson, 2004, 2005, 2008;
Dawson et al., 2009; Dawson et al., 2010).
Perceptrons are trained using an error-correcting variant of Hebb-style learning
(Dawson, 2004). Perceptron training associates input activity with output unit
error as follows. First, a pattern is presented to the input units, producing output
Elements of Connectionist Cognitive Science 189