Mind, Body, World- Foundations of Cognitive Science, 2013a

the only time that output unit activity will equal 1 is when both input units are activated
with 1 (i.e., when p and q are both true). This is because this situation will produce
a net input of 2, which exceeds the threshold. In all other cases, the net input
will either be 1 or 0, which will be less than the threshold, and which will therefore
produce output unit activity of 0.
The ability of the Figure 4-3 perceptron to compute AND can be described
in terms of the pattern space in Figure 4-2A. The threshold and the connection
weights of the perceptron provide the location and orientation of the single straight
cut that carves the pattern space into decision regions (the dashed line in Figure
4-2A). Activating the input units with some pattern presents a pattern space location
to the perceptron. The perceptron examines this location to decide on which
side of the cut the location lies, and responds accordingly.
This pattern space account of the Figure 4-3 perceptron also points to a limitation.
When the Heaviside step function is used as an activation function, the perceptron
only defines a single straight cut through the pattern space and therefore
can only deal with linearly separable problems. A perceptron akin to the one illustrated
in Figure 4-3 would not be able to compute XOR (Figure 4-2B) because the
output unit is incapable of making the two required cuts in the pattern space.
How does one extend computational power beyond the perceptron? One
approach is to add additional processing units, called hidden units, which are intermediaries
between input and output units. Hidden units can detect additional features
that transform the problem by increasing the dimensionality of the pattern
space. As a result, the use of hidden units can convert a linearly nonseparable problem
into a linearly separable one, permitting a single binary output unit to generate
the correct responses.
Figure 4-4 shows how the AND circuit illustrated in Figure 4-3 can be added
as a hidden unit to create a multilayer perceptron that can compute the linearly
nonseparable XOR operation (Rumelhart, Hinton, & Williams, 1986a). This perceptron
also has two input units whose activities respectively represent the state of
the propositions p and q. Each of these input units sends a signal through a connection
to an output unit; the figure indicates that the weight of each connection is 1.
The threshold of the output’s activation function () is 0.5. If we were to ignore the
hidden unit in this network, the output unit would be computing OR, turning on
when one or both of the input propositions are true.
However, this network does not compute OR, because the input units are also
connected to a hidden unit, which in turn sends a third signal to be added into the
output unit’s net input. The hidden unit is identical to the AND circuit from Figure
4-3. The signal that it sends to the output unit is strongly inhibitory; the weight of
the connection between the two units is –2.
Elements of Connectionist Cognitive Science 145