Mind, Body, World- Foundations of Cognitive Science, 2013a

Many other activation functions exist. One review paper has identified 640 different
activation functions employed in connectionist networks (Duch & Jankowski,
1999). One characteristic of the vast majority of all of these activation functions is
their nonlinearity. Connectionist cognitive science is associationist, but it is also
nonlinear.
4.4 The Connectionist Sandwich
Both the McCulloch-Pitts neuron (McCulloch & Pitts, 1943) and the perceptron
(Rosenblatt, 1958, 1962) used the Heaviside step function to implement the all-ornone
law. As a result, both of these architectures generated a “true” or “false” judgment
about each input pattern. Thus both of these architectures are digital, and
their basic function is pattern recognition or pattern classification.
The two-valued logic that was introduced in Chapter 2 can be cast in the context
of such digital pattern recognition. In the two-valued logic, functions are computed
over two input propositions, p and q, which themselves can either be true or false. As
a result, there are only four possible combinations of p and q, which are given in the
first two columns of Table 4-1. Logical functions in the two-valued logic are themselves
judgments of true or false that depend on combinations of the truth values of
the input propositions p and q. As a result, there are 16 different logical operations
that can be defined in the two-valued logic; these were provided in Table 2-2.
The truth tables for two of the sixteen possible operations in the two-valued
logic are provided in the last two columns of Table 4-1. One is the AND operation
(p·q), which is only true when both propositions are true. The other is the XOR
operation (pq), which is only true when one or the other of the propositions is true.
p q pq p q
1 1 1 0
1 0 0 1
0 1 0 1
0 0 0 0
Table 4-1. Truth tables for the logical operations AND (pq) and XOR (p q), where
the truth value of each operation is given as a function of the truth of each
of two propositions, p and q. ‘1’ indicates “true” and ‘0’ indicates “false.” The
logical notation is taken from McCulloch (1988b).
That AND or XOR are examples of digital pattern recognition can be made more
explicit by representing their truth tables graphically as pattern spaces. In a pattern
142 Chapter 4