Mind, Body, World- Foundations of Cognitive Science, 2013a

function such as the logistic. “No more than three layers are required in perceptronlike
feed-forward nets” (Lippmann, 1987, p. 16).
When output unit activity is interpreted digitally—as delivering “true” or “false”
judgments—artificial neural networks can be interpreted as performing one kind of
task, pattern classification. However, modern networks use continuous activation
functions that do not need to be interpreted digitally. If one applies an analog interpretation
to output unit activity, then networks can be interpreted as performing a
second kind of input-output mapping task, function approximation.
In function approximation, an input is a set of numbers that represents the
values of variables passed into a function, i.e., the values of the set x 1
, x 2
, x 3
, . . . x N
. The
output is a single value y that is the result of computing some function of those variables,
i.e., y = f(x 1
, x 2
, x 3
, . . . x N
). Many artificial neural networks have been trained to
approximate functions (Girosi & Poggio, 1990; Hartman, Keeler, & Kowalski, 1989;
Moody & Darken, 1989; Poggio & Girosi, 1990; Renals, 1989). In these networks, the
value of each input variable is represented by the activity of an input unit, and the
continuous value of an output unit’s activity represents the computed value of the
function of those input variables.
A system that is most powerful at approximating functions is called a universal
function approximator. Consider taking any continuous function and examining a
region of this function from a particular starting point (e.g., one set of input values)
to a particular ending point (e.g., a different set of input values). A universal function
approximator is capable of approximating the shape of the function between
these bounds to an arbitrary degree of accuracy.
Artificial neural networks can approximate functions. How well can they
do so? A number of proofs have shown that a multilayer perceptron with two
layers of connections—in other words, a single layer of hidden units intervening
between the input and output layers—is capable of universal function approximation
(Cotter, 1990; Cybenko, 1989; Funahashi, 1989; Hartman, Keeler, & Kowalski,
1989; Hornik, Stinchcombe, & White, 1989). “If we have the right connections from
the input units to a large enough set of hidden units, we can always find a representation
that will perform any mapping from input to output” (Rumelhart,
Hinton, & Williams, 1986a, p. 319).
That multilayered networks have the in-principle power to be arbitrary pattern
classifiers or universal function approximators suggests that they belong to the
class “universal machine,” the same class to which physical symbol systems belong
(Newell, 1980). Newell (1980) proved that physical symbol systems belonged to this
class by showing how a universal Turing machine could be simulated by a physical
symbol system. Similar proofs exist for artificial neural networks, firmly establishing
their computational power.
Elements of Connectionist Cognitive Science 151