Mind, Body, World- Foundations of Cognitive Science, 2013a

The multilayer network illustrated in Figure 4-4 is atypical because it directly
connects input and output units. Most modern networks eliminate such direct connections
by using at least one layer of hidden units to isolate the input units from
the output units, as shown in Figure 4-5. In such a network, the hidden units can
still be described as carving a pattern space, with point coordinates provided by the
input units, into a decision region. However, because the output units do not have
direct access to input signals, they do not carve the pattern space. Instead, they
divide an alternate space, the hidden unit space, into decision regions. The hidden
unit space is similar to the pattern space, with the exception that the coordinates of
the points that are placed within it are provided by hidden unit activities.
Output
Unit
Modifiable
Connections
Hidden
Units
Modifiable
Connections
Input
Units
Figure 4-5. A typical multilayer perceptron has no direct connections between
input and output units.
When there are no direct connections between input and output units, the hidden
units provide output units with an internal representation of input unit activity.
Thus it is proper to describe a network like the one illustrated in Figure 4-5 as being
just as representational (Horgan & Tienson, 1996) as a classical model. That connectionist
representations can be described as a nonlinear transformation of the input
unit representation, permitting higher-order nonlinear features to be detected, is
why a network like the one in Figure 4-5 is far more powerful than one in which no
hidden units appear (e.g., Figure 4-3).
When there are no direct connections between input and output units, the representations
held by hidden units conform to the classical sandwich that characterized
Elements of Connectionist Cognitive Science 147