Mind, Body, World- Foundations of Cognitive Science, 2013a

if they are weakly equivalent), accomplish this with the same algorithm, and bring
this algorithm to life with the same architecture. Cognitive scientists are in the business
of making observations that establish the strong equivalence of their models
to human thinkers.
Classical cognitive science collects these observations by measuring particular
behaviours that are unintended consequences of information processing, and which
can therefore reveal the nature of the algorithm that is being employed. Newell
and Simon (1972) named these behaviours second-order effects; in Chapter 2 these
behaviours were called artifacts, to distinguish them from the primary or intended
responses of an information processor. In Chapter 2, I discussed three general
classes of evidence related to artifactual behaviour: intermediate state evidence,
relative complexity evidence, and error evidence.
Note that although similar in spirit, the use of these three different types of
evidence to determine the relationship between the algorithms used by model and
subject is not the same as something like the Total Turing Test. Classical cognitive
science does not require physical correspondence between model and subject.
However, algorithmic correspondences established by examining behavioural artifacts
put much stronger constraints on theory validation than simply looking for
stimulus-response correspondences. To illustrate this, let us consider some examples
of how intermediate state evidence, relative complexity evidence, and error evidence
can be used to validate models.
One important source of information that can be used to validate a model is
intermediate state evidence (Pylyshyn, 1984). Intermediate state evidence involves
determining the intermediate steps that a symbol manipulator takes to solve a problem,
and then collecting evidence to determine whether a modelled subject goes
through the same intermediate steps. Intermediate state evidence is notoriously difficult
to collect, because human information processors are black boxes—we cannot
directly observe internal cognitive processing. However, clever experimental paradigms
can be developed to permit intermediate states to be inferred.
A famous example of evaluating a model using intermediate state evidence
is found in some classic and pioneering research on human problem solving
(Newell & Simon, 1972). Newell and Simon collected data from human subjects as
they solved problems; their method of data collection is known as protocol analysis
(Ericsson & Simon, 1984). In protocol analysis, subjects are trained to think out loud
as they work. A recording of what is said by the subject becomes the primary data
of interest.
The logic of collecting verbal protocols is that the thought processes involved in
active problem solving are likely to be stored in a person’s short-term memory (STM),
or working memory. Cognitive psychologists have established that items stored in
such a memory are stored as an articulatory code that permits verbalization to
98 Chapter 3