JUDGMENT-BASED AND REASONING-BASED STOPPING RULES ...

JUDGMENT-BASED AND REASONING-BASED STOPPING RULES
IN DECISION MAKING UNDER UNCERTAINTY
Kathryn Ritgerod Nickles
Wake Forest University
Shawn P. Curley
University of Minnesota
P. George Benson
Rutgers University
Working Paper. Please do not quote without permission.
Send correspondence to
Kathryn Ritgerod Nickles
The Wayne Calloway School of Business and Accountancy
Wake Forest University
P.O. Box 7285 Reynolda Station
Winston-Salem. NC 27109-7285
(910) 759-4410
FAX (910) 759-6133
nicklesk@wfu.edu
December 4, 1995
1

AB.S'T RA CT
As a practical matter, people necessarily use some kind of stopping criterion to take
actions in the world. Frequently, stopping is dictated by intema/factors or aspects of the
decision maker's thinking. One such aspect is the individual's internal assessment of whether
enough information has been obtained. This research investigates the bases for this assessment
and asks from a cognitive perspective: What underlies the individual's assessment that she has
obtained enough information to draw a conclusion?
This question is addressed within the context of probability assessment. Two key mental
activities, judgment and reasoning. are differentiated and provide the foundation for categorizing
proposed stopping criteria in probability assessment. Judgment-based and reasoning-based
stopping rules are identified, and their predictions for stopping are tested empirically. The
methodology consists of deferred decision-making (optional stopping) tasks combined with "think
aloud" verbal protocols.
Whereas previous research acknowledged stopping, if at all, only in terms of judgment or
as ad hoc rules, this research highlights the role of reasoning in stopping. In particular, the
reasoning-based mental list role is implicated. With this rule, it is the individual's internal list of
requisite components for task completion that serves as the stopping criterion.
The research also
implicates, but to a lesser extent, the judgment-based magnitude threshold role. With this rule, it
is the individual's cumulative assessment of the evidence as haVing reached or exceeded an
internally held threshold that influences stopping.
KEYWORDS decision analysis, probability assessment, mental representation,
reasoning, judgment, stopping rule, deferred decision-making.
2

1. INTRODUCTION
Stopping rules are a fundamental aspect of human thinking. As a practical matter, people
necessarily use some type of stopping criterion in order to initiate actions in the world. The
consumer, for example, determines that he has obtained enough information about a particular
used car, and he stops his strategy of shopping around. The physician completes a review and
assessment of test results and diagnoses the patient's illness. The economist shifts from data
collection to the construction of a forecast of next year's housing starts based on her knowledge
and the available data.
In each of these situations. external factors such as the time or monetary resources
available to the decision maker may determine when conclusions are drawn. Often, however, an
external factor is not critical. Instead, stopping is dictated by internal factors or aspects of the
decision maker's thinking. For example, the housing forecaster may judge that she has obtained
enough information about current economic conditions. Rather than time or monetary
constraints, it is her internal state that functions as the stopping criterion, prompting her to move
from data collection to conclusion.
Of particular relevance to stopping criteria are the empirical findings that have consistently
suggested that individuals frequently stop prematurely in decision tasks, for example, by adopting
the first plausible conclusion. People may use naive reasoning to accept a claim that "sounds
right," minimizing cognitive effort but sacrificing accuracy {Perkins, Allen, & Hafner, 1983, p.
186). Researchers point out that errors in decision making often occur because individuals stop
too soon (Baron, Beattie, & Hershey, 1988). People fail to consider alternatives (Farquhar &
Pratkanis. 1993). Decision makers fail to access relevant information, and they leave out or
overlook important elements (Fischhot'r, 1977; Fischhot'r, Slovic, & Lichtenstein. 1978;
Shafir & Tverksy, 1992).
3

!fit is the individual's internal assessment of having obtained enough infonnation that
prompts her to stop, then the bases for that assessment should be investigated. The current
research asks fi"om a cognitive prospective
What underlies the individual's assessment that she
has obtained enough information to draw a conclusion? Asked another way What aspects of the
individual's thinking activities prompt her to stop accessing available information?
These questions are addressed within the broad context of decision making under
uncertainty with a focus on probability assessment. Specifically, this research investigates the
internal stopping rules that prompt the decision maker's personal assessment that she has enough
information to provide the probability that an event will occur. This context was chosen because
of the critical role played by probability assessment in the predominant theory of decision making
under uncertainty (von Newmann & Morgenstern. 1947; von Winterfeldt & Edwards, 1986) and
because of its role in forecasting tasks (e.g., Murphy & Winkler, 1984).
In general, little attention has been paid to stopping phenomena (Nickles, 1995). In
decision analysis, checks for consistency and stability of the decision maker's responses are taken
as evidence that stopping is appropriate (Spetzler & Stael von Holstein, 1975). These ad hoc
rules have no connection to the thinking processes used by the assessor in the construction of
probability or preference assessments. Other stopping criteria. discussed explicitly in the study of
sequential or deferred decision making, include rules that have a specified theoretical connection
to normative models of stopping (Busemeyer & Rapoport, 1988; Peterson & Beach, 1967; Pitz.
Reinhold, & Geller, 1969). Typically, researchers employ implicit assumptions about the decision
maker's mental activities including his mental scaling and weighing activities and the ability to set
and maintain mental standards for making comparisons. However. no clear cognitive perspective
is provided within which to address these implicit assumptions about aspects of the decision
maker's thinking that may influence the individual to stop seeking information. Moreover, the
4

particular scale or construct that is presumed to be monitored by the decision maker is generally
ill-defined.
The current research addresses these deficiencies.The paper uses a cognitive framework
for the mental activities involved in probability assessment that provides the foundation for
identifying and categorizing proposed stopping criteria in probability assessment. These criteria
are then tested empirically. The methodology and analyses are described. Finally, a discussion
explains the implications of these findings for stopping prematurely in probability assessment.
2. STOPPING RULES IN PROBABILITY A.S'SESSMENT: A COGNITIVE PERSPECTIVE
The traditional cognitive framework for probability assessment posits that the decision
maker, drawing on available evidence, uses judgment, which is assumed to be some type of mental
evaluation, to produce the probability (Hogarth, 1987). This type of representation has been
implicit in the research discussed above in which stopping criteria have been studied (Busemeyer
et aI., 1988; Pitt et aI., 1969)
In contrast, a more detailed framework for understanding what the individual actually does
cognitively derives from recent accounts of probability assessment that emphasize the constructive
nature of probability (Curley & Benson, 1994; Payne, Bettman. & Johnson, 1992; Shafer, 1981;
Smith, Benson. & Curley. 1991). According to this view, the essence of probability assessment is
the individual's attempt to form a belief about the occurrence of the event of interest, a process
dominated by his cognitive capacity to reason. Drawing on available evidence and using
reasoning, the individual constructs arguments to reach a conclusion. This part of the assessment
process, which is so dominated by informal reasoning, has been described as belief assessment
(Benson, Curley, & Smith, in press). Subsequently, the assessor determines his degree of belief in
the conclusion, culminating in a probability. This latter phase is referred to as response
assessment; its characteristic activity is judgment, defined as a mental evaluation or comparing
activity.
5

Identifying the two key cognitive activities of probability assessment provides a
framework for characterizing and developing stopping criteria. In past research, the explicit
internal stopping rules have all been judgment-based. For the judgment-based rules, the decision
maker is assumed to set and maintain some type of mental threshold along a key dimension, e.g.,
plausibility, and to keep a "running total" or measure relative to this dimension (e.g, Gettys &
Fisher, 1979). When the measure crosses the threshold, he ceases to access additional
inforntation, prompting action and completion of the task. Two general forms of judgment-based
stopping rules can be identified from the literature. depending on the nature of the threshold being
applied.
2lL Judgment-Based Stopping Rules. Using the judgment-based magnitllde threshold
role, the individual maintains an internal running total of the cumulative impact of the evidence in
support ora particular conclusion (e.g., a prediction that housing sales will increase) When the
cumulative support matches or exceeds the threshold, he ceases to collect additional evidence.
With this rule, the individual behaves as if the threshold for cumulative impact is the stopping
criterion.
Using the judgment-based rule called the difference threshold role, the individual assesses
the marginal value of the latest piece of evidence.The individual is assumed to keep an internal
record of the cumulative impact of accessed data along a key dimension (e.g., plausibility). In
addition, he is assumed to compare his cumulative assessment after the most recently accessed
infonnation item to that preceding the current item. When the absolute difference between these
two assessments falls below the individual's internal difference threshold, he stops. Intuitively, the
individual stops in the belief that he is learning nothing new; the evidence is not substantial enough
to influence his conclusion.
Using either of these judgment-based rules, the individual is assumed to evaluate evidence
on the basis of a key dimension, for example, plausibility, but the dimension is often ill-specified
6

or undefined. To achieve generality for the current study, an experimental procedure was
developed that did not require specification of the dimension being monitored in the rules. It is
discussed later in the paper.
lb. Reasoning-Based Stopping Rules. During the belief assessment phase of probability
assessment, as the individual collects evidence and constructs arguments using available
information, he develops a mental representation of the elements of the decision situation (Yates,
1990). These elements may include the arguments he constructs during informal reasoning,
previously constructed arguments, data gathered from external sources and/or evoked from long
term memory, and propositions under consideration.Three stopping rules that depend on the
individual's decision representation are hypothesized below. The rules are called reasoning-based
because each is dominated by the individual's informal reasoning processes.
When a person reasons informally during probability assessment, he generates arguments
that serve to develop and elaborate his conception of the decision situation. As he continues to
reason, some arguments may be relegated to long term memory due to the limited capacity of
working memory, while other arguments move into working memory. For example, his attention,
and thus his mental representation. may spotlight only the most recently constructed arguments
(Zechmeister & Nyberg, 1982). As the individual accesses information, constructs arguments,
and accommodates his mental representation, he at some point may return to a prior internal
representation containing the same subset of arguments. Returning to the same subset of
arguments can be interpreted as indicating a stability of the problem representation. This then can
be used as a signal to terminate the development of his internal representation, a phenomenon
referred by Yates and Carlson (1982) as representational stability. The representational stability
nile suggests that it is a recurring representation that influences the individual to stop accessing
additional information and to reach a conclusion.
"7

The second reasoning-based stopping rule is similar to the first rule in that stopping is tied
to stability within the mental representation. In this case, the individual may tentatively form
conclusions as he is accessing infonnation. As he constructs arguments, at some point his
conclusion may achieve stabiJity. The propositional stability nile suggests that it is the
unchanging nature of the decision makers conclusion that prompts stopping.
Using the mernallist nile, when asked to reach a conclusion, the individual develops or
accesses from long term memory a mental list of the components he believes are required for a
response. Belief structures such as schemas (Bartlett. 1932) or scripts (Schank & Abelson, 1977)
may guide and organize the individual's construction of the mental list that becomes the set of
criteria for task completion. Once the decision maker believes he has collected information on the
requisite elements (collects the evidence and/or constructs the arguments), he reaches a
conclusion and completes the task.
3. METHODOLOGY
Stopping Rules' Predictions. The five identified stopping rules were tested using pairs of
contrasting situations in three experiments. The study's tasks were constructed to have ecological
validity with the goal that any results should generalize to real-world probability forecasting.
Specifically, participants were asked to provide probability assessments when predicting sales in
the national housing market and when predicting the short term interest rate of a small bank.
Conditions that individuals typically face in such tasks were manipulated in the contrasting
situations within the three experiments.The three contrasts included providing the forecaster
with (1) familiar or repeated information versus all new, non-repeated information, (2)
information items that regularly alternate in support of different conclusions versus unanimous
data that support a particular conclusion. and (3) contradictory infonnation versus confinning
information relative to the individual's initial conclusion.
8

The different rules offered different predictions in the contrasting situations. Table 1
provides a summary of these predictions.The column headings in the table provide the contrasts
afforded by the information manipulations. Each row of the table corresponds to a different
stopping rule. Each cell in the table represents a particular stopping rule's prediction in light of
the specific manipulation of the data. The next three sub-sections provide the rationales
underlying the various predictions.
[Insert Table here. ]
Familiar (F) l'S. Netll (N). When constructing a probability assessment, an individual may
access information that is essentially the same as data seen previously, i.e., familiar information. If
the individual is monitoring the cumulative impact ora key variable (e.g., plausibility), such data
are likely to have little effect on this measure. Consequently, the two threshold rules would
predict that accessing familiar information items would be evaluated by the forecaster as less
useful for the task. For the magnitude threshold rule. this means that the threshold is less likely to
be crossed, and later stopping is likely to occur compared to the assessor who sees all new,
non-repeating information. This prediction is summarized as F > N in Table 1
According to the difference threshold rule, however, familiar information implicates a
small difference between the current set of information items and the previous set. The small
difference could very well fall below the difference threshold. leading to stopping. Thus,
according to the difference threshold rule. and in contrast to the magnitude threshold rule, earlier
stopping is likely when available data are familiar, that is, F < N.
For the representational stability rule, familiar data are likely to lead to the construction of
arguments similar to arguments generated earlier or to the recall of previous arguments. A
continued focus on essentially the same set of arguments prompts the participant to stop and
reach a conclusion, that is, F < N
9

Propositional stability similarly predicts earlier stopping, i.e., F < N. As the decision
maker accesses familiar information, her conclusion is likely to remain the same. According to
this rule, such stability prompts completion of the task.
In contrast, the mental list rule does not predict earlier stopping when data are familiar.
Presumably, familiar information items are ones that already have been compared to the
individual's list. Since other items are still being sought, she should continue, that is, F > N.
Alternating (A) vs. Unanimolls .~upport (U). The next pair of contrasts operates
similarly, although the pair is designed to distinguish different sets of stopping rules.
infonnation manipulation provides the individual with data that regularly alternate in support of
different conclusions (A) or data that are unanimous in support of one conclusion (U).
distinguish predictions among the stopping rules and to control for the possible influence of
familiar data. only non-repeating items are made available.
According to the magnitude threshold rule, it is the individual's cumulative assessment of
information in support of a particular conclusion that influences stopping. When available data
vary systematically in terms of their support of alternative conclusions, it should take longer to
cross either threshold since movement towards one threshold is immediately followed by
movement towards the opposite threshold. As a consequence, the individual is expected to access
more information before her critical threshold for a particular conclusion is reached compared to
when all data are unanimous for a particular conclusion.The prediction is A > U in Table 1
For the difference threshold rule, only the absolute value of the differences is compared to
the difference threshold; directionality is not relevant.The rule operates similarly whether
information items are provided in an alternating or unanimous fashion. The difference threshold
rule makes no prediction for differential stopping, that is, A = U.
The representational stability rule predicts differential stopping. Data that alternate in
support of different conclusions are likely to introduce new elements into the individual's mental
10

conception of the decision situation and prompt additional argument construction. In this way,
the individual continues to elaborate her internal representation and stability is unlikely to occur.
In contrast, when the data are unanimous, new arguments are less likely to arise, the decision
representation becomes stable, and earlier stopping occurs.
The prediction is A> U.
Likewise with the propositional stability rule, when information items alternate in support
of different conclusions, the assessor is less likely to maintain the same conclusion. In contrast,
unanimous information should lead to a stable conclusion sooner and earlier stopping, A > U.
The mental list rule does not predict differential stopping, that is, A = U. According to
this rule, the individual stops when she accesses all the elements she believes are required for
reaching a conclusion. The direction of the support of the inforntation items is not relevant.
Contradictory (C1) vs. Confirming (CF). The third information manipulation that
distinguishes predictions among the stopping rules involves providing contradictory evidence
(CT) to the individual or providing confirming evidence (CF) after an initial response is generated
To control for the possible influence offamiliar data, only non-repeating items are made available
to the participant. This manipulation allows participants to reach an initial conclusion before an
additional piece of evidence is introduced and the assessor is allowed to continue (or not) as
before.
Ifa participant used either of the judgment-based rules, a critical threshold would have
been reached in achieving the initial conclusion. For the magnitude threshold rule, contradictory
evidence, but not confirming evidence, could cause the individual to fall below the threshold. If
so, the assessor would continue to access additional infonnation until the threshold is again
crossed. In contrast, the individual who is provided with confirming evidence to an initial forecast
would not be expected to continue collecting evidence or to change her opinion. The prediction
is CT > CF in Table

Using the difference threshold rule, the judgment of the latest difference mayor may not
fall below her internal difference threshold, and she mayor may not continue to access data.
However. because this rule considers only the absolute difference of the individual's assessments
and not the contradictory or confirming nature (i.e., direction) of the data, it does not predict
differential stopping, that is, CT = CF.
The representational stability rule predicts differential stopping. Contradictory data are
likely to prompt the individual to generate new arguments that could be unrelated to those
generated earlier.' With new argument construction. the participant's internal representation is
enhanced, and representational stability is unlikely to occur. In contrast, continued elaboration of
the individual's internal representation is unlikely when confirming information is introduced.
Thus, the individual's internal representation remains stable, and she does not collect additional
evidence. that is, CT > CF.
The propositional stability rule does not predict differential stopping for contradictory and
confinning evidence that is introduced relative to the individual's initial response.When the
participant keeps her initial conclusion in the face of evidence that is contradictory (but not
definitively so), the conclusion appears robust. In this case, the ongoing stable quality of the
individual's conclusion prompts her to stop collecting evidence.The propositional stability rule's
prediction is CT = CF.
The mental list rule makes a similar prediction that differential stopping will not occur. If
contradictory evidence were introduced, it could change the value of the variable(s) on the
individual's intemaIlist but would not necessarily change the list itself. Stopping still occurs when
the person's set of elements for reaching a conclusion is met. The mental list rule predicts
CT=CF.
Research De.\'ign and Procedures. The research design for each of the three experiments
was a 2 X 2 within-subjects design in which 90 subjects, 30 in each experiment, participated in
12

deferred decision-making (optional stopping) tasks. Individuals were paid for their participation
and were recruited from among senior management students, MBA students, and other graduate
students. The main dependent variable in each experiment was the number of information items
accessed by participants. In addition. "think aloud" verbal protocols were collected (Ericsson &
Simon, 1993). In Experiment 3, a detailed analysis of subjects' protocols was conducted to
investigate the individual differences implicated in that study.
The first experimental factor was the manipulation of the information provided to
participants.The second factor was the business setting in which the forecasting tasks were
operationalized. either predicting housing sales or predicting the interest rate of a small bank.
Two versions of each setting were constructed so that each subject completed a total of four
forecasting tasks. two in each experimental condition. In each experiment. the order of
treatments was randomized within subjects with the exception that the two types of business
settings alternated with one another.
The researcher worked individually with each participant and the same general procedures
were followed in each experiment. Participants were instructed to imagine themselves as seeking
a position on a research staff. Their hiring evaluation required their completing several
forecasting tasks. The participant initially read aloud a short business scenario describing the
business situation in the task. After the initial information and after every subsequent piece of
information, the subject had the option to make a prediction or to defer reaching a conclusion in
order to obtain additional information, one item at a time, supplied by the researcher. This
deferred decision-making arrangement enabled control of each participant's access to information.
Participants were instructed that a timely forecast would be expected in each task, and
without sacrificing accuracy. they should provide their predictions as soon as possible.
Participants could request additional infonnation from the researcher, but a time tradeoff was
involved. A time lapse of 2 days was involved for each information item requested. Participants
1

were also informed that no predict~on would be considered "right" or "wrong." To avoid
motivational biases, no payoff or penalty was associated with any prediction made by participants.
Before beginning the four tasks. each participant was given a practice forecasting task that
required a prediction about the trend in retail sales of a clothing store. The practice task involved
the same procedures as described above.
4. DA T A ANAL YSES
4.1 E.~periment 1. In Experiment 1, each participant completed 4 tasks, two housing
sales predictions (H) and two interest rate predictions for a bank (B). In each setting, participants
were provided with either familiar, repeated data (F) or all new, non-repeating (N) information.
The Analysis of Variance, conducted with a General Linear Models Procedure, indicates main
effects from both factors: information manipulation (F (1, 87) = 7.13, P = .009) and setting
(F(l. 87) = 30.98.p = .0001), with no significant interaction between the two factors
(F(l, 87) = .99, P - .3214). Participants accessed on average more infonnation items when
provided familiar. repeated information than when provided new. non-repeated items. In addition,
participants accessed more information items in the bank interest rate setting compared to the
housing sales setting.2
The distributions of the paired differences for each setting/information manipulation in
Experiment 1. i.e.. (HF-HN) and (BF-BN). are represented graphically in Figure la. This display
shows that the majority of participants accessed more information when provided familiar,
repeated information, i.e., HF-HN >0 and BF-BN >0. This conclusion is also supported by a
nonparametric, one-tailed sign test which rejected the null hypothesis that the median difference
equals 0 (HF-HN: 18/25,p = .O216~ BF-BN:19/28,p = .0436). Significantly more items were
accessed than would be expected by chance when the data were familiar (F) versus new (N) in
both settings. These results implicate the magnitude threshold and mental list rules (Table 1).3
14

[Insert Figure here.]
The experimental methodology did not assume that participants necessarily would respond
similarly, i.e., use the same stopping rules, or that individuals necessarily would use the same
stopping rule in each prediction setting. However. the data allow the examination of the
consistency of participants' responses across both settings. The data in Table 2 provide an
illustration of the responses. Row 1 contains the number of participants who accessed more
information items in the bank interest rate setting in the familiar condition (BF-BN > 0); row 2
contains the number of participants who accessed the same number of items (BF-BN = 0); row 3
contains the number of participants who accessed more data in the new, nonrepeated condition in
the bank interest rate setting (BF-BN< 0). The columns are defined similarly for the housing sales
setting. As indicated by the shading in Table 2, almost half of the participants (13/30) responded
in a consistent manner with responses from 1
participants falling in the greater-than-zero
category for both settings, i.e., HF-HN> 0 and BF-BN > o. In summary, the results in
Experiment 1 generally appear to be consistent with the predictions of the magnitude threshold
and mental list stopping rules, i.e., HF > HN and BF > BN.
[Insert Table 2 here.
4.2 Experintent 2. In Experiment 2, thirty participants completed 4 tasks, two
predictions of housing sales (H) and two predictions ora bank's interest rate (B). In each setting,
participants were provided with either information items that regularly alternated in support of
different conclusions (A) or unanimous data that supported a particular conclusion (U).
The Analysis of Variance, conducted with a General Linear Models Procedure, indicated
that neither of the main effects was significant: information manipulation (F(l, 87) =
83,
p = .1798) or business setting {F (I, 87) = .02, P = .8809). No significant interaction between the
two factors was indicated (F(l, 87) = .02, P = .8809).
5

Figure Ib includes the graphical displays of the paired differences (HA-HU, BA-BU) of
participants' responses in the contrasting situations in both settings.
The mean differences in the
number of information items accessed in each setting are similar, and, in general, both plots are
evenly distributed around O. This conclusion is supported by a nonparametric, two-tailed sign test
which failed to reject the null hypothesis that the median difference equals zero (HA-HU: 12/23,
p - 1.0; BA-BU: 14/23, P = .4049). Thus, the outcomes in Experiment 2 do not support
significant differences, i.e., HA = HU and BA = BU.4
These results implicate the difference threshold and mental list stopping rules (Table 1).5
For these two rules, the direction of the information accessed by participants is not relevant. In
light of this aspect of the implicated rules and to corroborate the previous analysis, we examined
participants' sensitivity or lack of sensitivity to the direction of the evidence by comparing
participants' predictions to the evidence. For the difference threshold and mental list stopping
rules, one would expect as many predictions in one direction as in the other for the experimental
condition in which the data alternated in support of different conclusions. In fact, this occurred in
the bank setting. When participants were asked to predict a bank's future interest rate, 500/0 of
their conclusions (interest rate will increase or interest rate will decrease) matched the direction
indicated by the last item accessed, and 50% did not. However, this outcome is not replicated in
the housing sales setting. When participants were asked to predict future housing sales, 75% of
their conclusions matched the direction indicated by the last item accessed, and 25% did not. In
contrast, in the other condition when available data were unanimous in support of one conclusion
(U). almost all of the conclusions in either task matched the direction of the evidence accessed (as
expected under any rule) Thus. results in the bank interest rate setting further support
participants' use of the difference threshold and mental list rules to the extent that the direction of
the accessed information had no relevance.
16

The analysis of participants' responses across settings in Experiment 2 is provided in Table
3. As indicated by the shading in the table, only 9 participants responded in a strictly consistent
manner across business settings, less than the 13 consistent responses observed in Experiment 1
However, a clustering of responses about (0,0) occurred in Experiment 2 with almost half of the
responses (13/30) \vithin of (0.0) across settings. In summary. these results as well as those described
above appear generally to be consistent with the predictions of the difference threshold and mental
list stopping rules.
[Insert Table 3 here.
4.3 Experiment 3. Thirty participants in Experiment 3 completed a total of 4 tasks, two
predictions of housing sales (H) and two predictions ora bank's interest rate (B). In this
experiment, the information manipulation that distinguishes predictions among the stopping rules
involved providing contradictory evidence (CT) to participants versus providing confirming
evidence (CF) relative to their initial responses in the tasks.
The Analysis of Variance indicates main effects from the information manipulation
(F(I, 87) = 12.29,p = .0007), but not the business setting (F(l, 87) = .05,p = .8271), with no
significant interaction between the two factors (F{ 1,87) = .05; P = .8271). As indicated in Figure
lc, the mean difference in each setting (HCT -HCF. BCT -BCF) is small, less than one item.6
Nevertheless, participants on average accessed significantly more data in the contradictory
condition, i.e., when the available data contradicted their initial predictions.This conclusion is
supported by a non parametric, one-tailed sign test which rejected the null hypothesis that the
median difference equals 0 (HCT -HCF lO/IO.p = .OOI~ BCT-BCF: 9/11, P = .0327). As
expected, there are no significant differences in the mean number of items accessed by participants
before their initial predictions and. thus. before the information manipulation of either
contradictory or confirming evidence relative to their forecasts. In multiple tests, the observed
levels of significance werep > .25.
7

The graphical displays for Experiment 3 in Figure lc show that the responses fall mainly
into two groups. One category of responses, consistent with the overall statistical analysis above,
includes paired responses in which participants accessed more information when provided
contradictory evidence relative to their initial response, i.e., CT > CF. This category includes 10
pairs in the housing sales settings and 9 pairs in the bank interest rate setting.
These responses
implicate the magnitude threshold or representational stability rules (Table 1). The other category
includes responses in which the difference in the number of information items accessed in the two
contrasting situations is zero, i.e.. CT - CF In each setting, more than half of the paired
responses comprise this category and includes 20 pairs in the housing sales setting and 19 pairs in
the bank interest rate setting.
These responses are consistent with the difference threshold,
propositional stability, or mental list stopping rules (Table I).
There are 2 cases involving the bank interest rate setting in which two participants
accessed more information when the evidence confirmed their initial predictions compared to the
situation in which the available evidence was contradictory, i.e., BCT -BCF < o. These two cases
are anomalous results and are not predicted by any of the stopping rules. As such, they provide
an error rate against which to compare the responses in each of the two categories identified
above. For example. as a subset. the 41 responses that show either a zero difference or
less-than-zero difference could be expected to follow a binomial distribution. and, thus, fall
randomly into either category. However, for a binomial with n = 41 andp = .5, the cumulative
probability distribution function indicates that for paired differences in the less-than-zero category
as less than or equal to 2, the observed significance level is essentially zero (p < .00005). The
test, therefore, provides strong evidence that the 39 responses at zero (Figure lc) represent a
definite pattern. likewise, the J 9 responses that show a greater-than-zero difference greatly
outnumber the 2 less-than-zero responses (19/21, P = .0001
18

Comparison of individuals' responses across settings is provided in the 3 X 3
cross-tabulation that contains the frequencies of the less-than-zero, zero, and greater-than-zero
frequencies (Table 4). Consistent responses are indicated by the shading in the table. The modal
response in Experiment 3 was paired differences equal to zero. Almost half of the participants
(14/30 or 47%) accessed either no items following their initial predictions in either contrasting
condition (CT or CF) or the same number of items in both settings. i.e., HCT-HCF = 0 and
BCT-BCF - 0.'
[Insert Table 4 here.
In summary, the responses in Experiment 3 reveal two separate patterns. In one pattern,
participants accessed no information (or in one case the same amount of information) in both
contrasting situations after reporting their initial predictions. i.e.. CT = CF. These responses
implicate the difference threshold, propositional stability, or mental list rule (Table 1). In the
second pattern, participants sought more information when given a piece of contradictory
information than when given confirming information relative to their initial conclusions, i.e.,
CT > CF. These responses implicate the magnitude threshold or representational stability rule
(Table 1).
To investigate these individual differences further and to obtain more information about
the particular rules being implicated within these subgroups, a detailed analysis of subjects' verbal
protocols was conducted for Experiment 3. The following subsections describe, in turn, the
collection procedure, the coding schemes, and the data analyses.
4.3a ProtocolAnalysis. As participants completed each of the 4 prediction tasks, they
were asked to "think aloud," and report their thoughts as they considered the available data to
reach a conclusion (Ericsson & Simon. 1993) The verbal protocols were transcribed from audio
tape and parsed into conversational moves by one coder.. After the protocols were parsed, they
were coded using a system that was developed specifically to identify the participant's comments
19

as evidence for two key mental activities, judgment and reasoning.These codes were then
connected with the proposed judgment-based a.nd reasoning-based stopping rules. The system is
described in more detail below.9
Before coding was undertaken for the protocols in Experiment 3, a check was made for
the reliability of the coding scheme. Using the coding system. two coders independently scored
pilot subjects' verbal protocols and achieved an overall level of agreement of 84%. When
reliability was checked for distinguishing the three categories of judgment, reasoning, and "other,"
a level of agreement of92% was achieved between the coders. Subsequently, the verbal
protocols from Experiment 3 were scored by one coder.
4.3b Judgnlent Codes. An individual using a judgment-based stopping rule would be
expected to make comments consistent with the use of judgment in drawing conclusions. That is,
the comments should suggest the use of some type of mental scale and/or internal threshold to
gauge the available evidence.To categorize participants' comments that provide evidence of
judgment, a two-way classification was used; it is summarized in Table 5. The rows of the table
describe the distinction between an individual's assessment of the balance of evidence or direction
and an assessment of the weight of evidence or amount (Griffin & Tversky, 1992; Smith et aI.,
1991). The defining aspect for comments categorized in row is that the direction of evidential
impact is noted. For example, the individual may comment that he assessed the evidence as either
positive or negative in favor of a prediction for an increase in sales. In contrast, remarks were
coded in the second category (row 2) when they addressed the weight or credibility of the
evidence. In general, comments of this type provide an indication of the amount or absolute
strength of the evidence without an indication of the direction toward which the evidence points.
[Insert Table 5 here.]
The second dimension used for differentiating an individual's use of judgment is in tenns of
the number of information items addressed. An individual's judgmental assessment could relate to
20

a single information item, to a subset of the items such as in comparing one information item to
another, or to the evidence as a whole Table 5 provides a cell-by-cell categorization for
participants' comments that relate to the use of judgment and to the use of the judgment-based
stopping rules.
4.3c Mental Representation Calles. A participant's comments may contain evidence of
the use or content of an internal mental representation as she reached her conclusion via reasoning
in the sequential stopping tasks. Comments about the subject matter of the task would be
expected. Arguments and interconnections between newly accessed information and previously
obtained information would be expected. Such comments would reflect the makeup of the
individual's internal representation. Insight into the individual's internal representation connects to
the reasoning-based stopping rules, such as the representational stability, propositional stability, or
mental list rules, that employ such representations.
The six codes shown in Table 6 were developed to classify a subject's comments as
reflecting the content of her mental representation of the decision situation, These codes fall into
two general types: meta-statements and evidence of mental representation construction.
Meta-statements are statements concerning the individual's knowledge about the task and/or the
information she considers for completing it. For example, the code, MSwhole' was used for
meta-statements that revealed something of the subject's mental representation as a whole. The
code, MS., was used for the meta-statement suggesting a set of elements believed by the
individual to be important for completing the task. The third meta-statement code, MSr-iJilr' was
used when the individual recognized familiar information or acknowledged data as not previously
accessed.
[Insert Table 6 here.
The second general type of codes was used to mark statements indicating that the
individual is constructing or expanding her internal representation by adding elements or linkages
21

etween elements in the decision situation. When comments suggested that a linkage was being
made between items contained in the experimental task, code RC\aSL:. was used to mark this type of
representation construction. Interconnections that went beyond the content provided in the
experimental material were coded RCeXten.a, Tentative conclusions were coded RC~ as
evidence that a new element, i.e., the conclusion, had been added to the decision representation.
4.3d Results. Although a perfect mapping does not exist between the proposed stopping
rules and each code used in the verbal protocol analysis. certain correspondences would be
expected and are summarized in Table 7. The stopping rules are listed in the first column,
grouped under the two dominant patterns of responses in Experiment 3 that each implicate a
subset of stopping rules. The judgment and mental representation codes that are associated with
each rule are listed in the second column. For example. ifpal1icipants in the zero group were
using the mental list rule, more instances on average of code MSset would be expected in the zero
group of responses compared to the greater-than-zero group. This particular mental
representation code was used for participants' meta-statements that suggested a set of elements
believed to be important for completing the task, and, thus, connects to the mental list rule.
[Insert Table 7 here.
One-tailed test results indicate that more instances on average of the mental representation
codes, RC-'usKIII (I (37) = 91, P = .032) and MSSd (I (35) = 60; P = .059) occurred in the zero
group of responses compared to the greater-than-zero group. These outcomes are consistent
with use of the reasoning-based proposi~ionaJ stability and mental list rules. Both are implicated
by the zero group of responses.The zero group also implicated the difference threshold rule.
However, the data do not suggest a significant difference between the zero and greater-than-zero
groups in the average number of each of the three judgment codes. B-' WI, orW ~ ' associated
with the difference threshold rule. Even when these three judgment codes are combined, although
22

on average there are more instances in the zero group of responses than in the greater-than-zero
responses, the means do not differ significantly (I (37) = .5, P = .3
Mental representation codes RCIas1; and RCc.'(1cmaJ were used to mark comments to indicate
the expansion of the individual's internal representation. More instances of these codes in the
greater-than-zero group would have been consistent with use of the representational stability rule,
implicated by the greater-than-zero responses. However, there were more instances on average
of each of these codes in the zero group than in the greater-than-zero group
(RC-
t (36) = 37.p = .089; RCe.'UemaI: t (37) = 38, P = .089). The greater-thaD-zero
responses also implicated the magnitude threshold rule. For the judgment code, Ball' associated
with the magnitude threshold rule, the test results suggest that significantly more instances
occurred in the greater-than-zero group (I (21) - -1.40, P = .088). For the judgment code, Wall'
also consistent with the magnitude threshold rule but based on the weight rather than the balance
of the evidence. the means of the greater-than-zero and zero responses did not differ significantly.
Three codes, MSwIM>le, MS ram.li.. and B I . were not expected as diagnostic of any of the proposed
stopping rules. As expected. no significant differences obtained across groups (allp > .36).
Thus, the protocol analyses are consistent with the participants in the zero subgroup using
the propositional stability or mental list stopping rules. These are both reasoning-based rules. For
the greater-than-zero subgroup, the analyses are consistent with the magnitude threshold rule.
5. SUMMARY AND IMPLICATION.~ OF RESULTS
Each experiment was designed to implicate a subset of the proposed stopping rules as
suggested by the predictions in Table Table 8 provides a summary of the supported
implications. In Experiment I, participants accessed data in a manner consistent with the
predictions of the magnitude threshold and mental list stopping rules. In Experiment 2, the data
were consistent with the difference threshold and mental list stopping rules. Analysis of the data
collected in Experiment 3 revealed two subgroups of subjects showing different response
23

patterns. In the first subgroup, participants accessed more data when the data contradicted rather
than confirmed their initial predictions~ this is consistent with the magnitude threshold or
representational stability stopping rules. In the other subgroup, there was no effect of the
manipulation, implicating the difference threshold, propositional stability, or mental list rules.
[Insert Table 8 here.
An analysis of the verbal protocols collected in Experiment 3 from participants in the first
subgroup, those who accessed more data in the contradictory condition, pointed to the magnitude
threshold rule as operative for these subjects. More specifically, it was also suggested that the
scale being monitored in trusjudgment-based rule was based on the balance (direction), not the
weight (amount) of the evidence.The analysis of comments from participants in the second
subgroup, those who did not access additional information in either contrasting condition, pointed
to the use of the reasoning-based propositional stability and mental list rules for these subjects.
A particularly noteworthy overall finding is the consistent support across all three
experiments for the rero'oning-has'ed mental list rule. This finding expands our conception of the
stopping rules that subjects may use. Previous research involving deferred-decision tasks has
concentrated exclusively on judgment-based stopping rules. In contrast, the current study
provides support that stopping rules emanate from both judgment and reasoning. Accordingly,
the current study advances our understanding of how these mental activities may influence the
individual's completion of a probability assessment task and. thus. provides prescriptions for
avoiding premature stopping. For example, knowledge that the assessor uses a particular
reasoning-based rule may suggest measures for exploiting the informal reasoning activities that
dominate belief assessment, the initial phase of probability assessment. As the findings in
Experiments 2 and 3 demonstrate, if aspects of stopping are content or reasoning-based, then
relying on the introduction of contradictory evidence to prevent premature stopping may not be
effective. Instead, the content of the available data must prompt the individual to question her
24

epresentation of the decision situation. Otherwise, the inadequacy of the representation
underlying the individual's mental list may still cause her to stop accessing additional information
prematurely.
In addition, the current research holds implications for probability assessment and decision
making more generally.The prominence of reasoning-based rules supports the view that
individuals' cognitive capacity to reason plays a significant role in the construction of probability
assessments, expanding the more typical judgment-centered approach to probability assessment
and decision making.
Issues to be addressed in future research include the possibility that stopping criteria
operate simultaneously and/or in tandem The results from Experiment 1, for example, implicate
both the mental list rule and the magnitude threshold rule. The verbal protocol analysis in
Experiment 3 reveals prevalent patterns across contexts, information manipulations, and response
patterns of both reasoning and judgment codes that are associated with particular stopping rules.
Important to address in future research are the potential complexities and the possible interactive
effects when use of two or more of the stopping rules is indicated
In addition, results from the current study do not rule out the possibility that other
stopping rules may function in probability assessmentWide
individual differences were apparent
in each experiment
These differences may suggest stopping criteria not previously identified and
motivate continued investigation in this area.
25

REFERENCES
Baron. J., J. Beattie, and I.C. Hershey (J988). Heuristics and Biases in Diagnostic Reasoning:
Congruence, Information, and Certainty. Organizational Behavior and Human Decision
Processes, 42, 88-1 J O.
Bartlett, F. C. 1.932). Remembering. Cambridge: Cambridge University Press.
Benson, P.G., S.P. Curley, and G.F. Smith (in pres~'). Belief Assessment: An Underdeveloped
Phase of Probability Elicitation. Management Science.
Busemeyer, l.R., and A. Rapoport (1988). Psychological Models of Deferred Decision Making.
Journal of Mathematical Psychology, 32, 91-143.
Curley, S.P., and P.G. Benson (1994). Applying a Cognitive Perspective to Probability
Construction. In G. Wright and P. Ayton (Eds.), 5ilbjeclive Probability. Chichester,
England: John Wiley & Sons, 185-209.
Ericsson, K.A., and "H.A. Simon (1993). Protocol Analysis: Verbal Repor/s as Data.
Cambridge, MA: MIT Press.
Farquhar, P.H., and A.R. Pratkanis (1993). Decision Structuring with Phantom Alternatives.
Management L\'cience. 39, 1214-1226.
Fischhoff: B. (1977). Cost Benefit Analysis and the Art of Motorcycle Maintenance. Po/icy
Sciences, 8, 177-202.
Fischhoff, B., P. Slovic, and S. Lichtenstein (1978). Fault Trees: Sensitivity of Estimated Failure
Probabilities to Problem representation. JOt/mal of Experimental P~chology: Human
Perception and Performance, 4, 330-344.
Gettys, C.F., and S.D. Fisher, "Hypothesis Plausibility and Hypothesis Generation,"
Organizational Behavior and Human Perfornlance, 24 (1979), 93-110.
Griffin. D.. and A.Tversky (1992). The Weighing of Evidence and the Determinants of
Confidence. Cog1li1ivl! Psychology. 24. 411-435.
Hogarth. R. (1987). Jlldgmenl Gnd (~h(}ice: The Psychology of Decision. New York: John Wiley
& Sons.
Murphy, A.H., and R.L. Winkler (1984). Probability Forecasting in Meteorology. Jouma/ofthe
American Statistical A.~ociation. 79,489-500.
Nickles, K.R. (1995). Judgment-Based and Reasoning-Based Stopping Rules in Decision Making
Under Uncertainty. Doctoral Dissertation, University of Minnesota. Dissertation
Abstracts International, 56, IOO5A. (Order #DA952 1562)
26

Payne, J.W.. J.R. Bettman. and E.J. Johnson (1992). Behavioral Decision Research: A
Constructive Processing Perspective. Al/mfal ReviL.l1v of P.sychology, 43, 87-131
Perkins, D.N., R. Allen, and J. Hafner (1983). Difficulties in Everyday Reasoning. In W.
Maxwell (Ed.), Thillki/lg: 711e Expalldi/lg Frontier. Philadelphia: Franklin University
Press.
Peterson, C.R., and L.R. Beach (1967). Man as an Intuitive Statistician. P.sychologicalBulletin,
68, 29-46.
Pitt, G.F., H. Reinhold, and E.S. Geller (1969). Strategies of Information Seeking in Deferred
Decision making. Organizational Behavior and HlIman Performance, 4, 1-19.
Reichman-Adar. R. (1984). Extended Person-Machine Interface. Artificial Intelligence, 22,
157-218.
Schank, R.C., and R. P. Abelson (1977). Script.\'. Plans, Goals, and Understanding: An Inquiry
into Humml Knowledge Stnlchlres. Hillsdale, NJ: Erlbaum Associates.
Shafer, G. (1981). Constructive Probability. Syntht'st', 48, 1-60.
Shafir, E., and A. Tversky (1992). Thinking through Uncertainty: Nonconsequential Reasoning
and Choice. Cogl1itive P.sychology. 24,449-474.
SpetzJer, C.S., and C.S. Stael von Holstein (1975). Probability Encoding in Decision Analysis.
Management Science. 22,340-358.
Smith, G.F., P.G. Benson, and S.P. Curley (1991). Belief, Knowledge, and Uncertainty: A
. Cognitive Perspective on Subjective Probability. Organizational Behavior and Human
Decision Processes. 48, 291-321.
von Neumann, J., and O. Morgenstern (1947). Theory of Games aJldEconomic Behavior.
Princeton, NJ: Princeton University Press.
von Winterfe1dt, D., and W. Edwards (1986). Decision Analysis and BehmlioralResearch.
Cambridge: Cambridge University Press.
Yates, I.F. (1990). Jlldgment O/ld Deci~'ionMaking. Englewood Cliffs, NI: Prentice-Hall.
Yates, I.F., and B.W. Carlson (1982). Toward a Representational Theory of Decision Making.
Ann Arbor, MI: University of Michigan, WorkingPaper.
Zechmeister, E.B., and S.E. Nyberg (1982). Human Memory: An Introduction to Research and
Theory. Monterey, CA: Brooks/Cole.
27

FOOTNOTES
I In previous research, decision makers who were instructed to generate an opposing
argument to an initial conclusion often generated a new line of arguments that in some cases led
to different conclusions (perkins et aJ., 1983).
2 During data collection. participants in Experiment 1 frequently commented on their lack of
knowledge with regard to forecasting future bank interest rates and often appeared to be more
comfortable predicting future housing sales in the national housing market. Of the 18 instances in
which participants assessed a probability of .5, a possible indication that their predictions were
pure guesses, all occurred in the bank interest tasks, 17 in the familiar condition (F) and 1 in the
new data condition (N).
3 Of the remaining cases of nonzero differences, 16 were in the negative direction,
(F - N < 0) consistent with the predictions of the difference threshold, representational stability, or
propositional stability rules. Compared to the anomalous zero differences, (F - N = 0), there were
7 housing sales responses (H: 7/12; I-tailedp = .3872) and 9 bank interest responses (B: 9/11; 1
tailed p = .0327) in which more items were accessed in the new, non-repeating condition (N).
Thus, across settings, strong evidence does not exist for a secondary minority response pattern.
4 The plots for Experiment 2 in Figure I show one outlier in the housing sales setting
(HA-HU = 10). However, exclusion of this extreme value from S9 did not change the conclusion
and so it was included in the analyses.
S As a subset, the 20 instances of nonzero differences in the negative direction in Figure 1,
i.e., HA-HU < 0 and BA-BU < 0, are anomalous, not predicted by any of the stopping rules.
However, 13 responses in the negative direction equal -I and may be random variation along with
the 11 responses equal to + I. Including these + 1 or -1 responses, the clustering of the data
around zero, i.e., 16 of30 responses in the housing sales setting and 22 of30 in the bank interest
rate setting, supports the conclusion of HA = HU and BA = BU as the predominant response
pattern.
6 The graphs of the distributions of the paired differences for the settings (HCT -HCF ,
BCT -BCF) in Experiment 3 reveal three outliers, values more than three standard deviations from
the mean. Two of these were generated by one individual (HCT -HCF = 5, BCT -BCF = 7). The
third outlier was generated in the housing sales setting (HCT -HCF = 4). Significance is
maintained after these three values are excluded (HCT -HCF: 8/8, p = .0039; BCT -BCF: 8/10,
p = .0547), and, therefore, these three data points are included in the analyses.
7 523 was the only participant who accessed the same number of information item in each
of the housing sales tasks, one item in each contrasting situation, i.e., HCT-HCF = 0, and,
therefore, is included in the 4?O1o.
8 A conversational move is the smallest segment of an individual's utterance that makes
sense (Reichman-Adar, 1984). It is a distinguishable phrase or sentence that holds meaning such
that any further division risks destruction of the expression's content.
28

FOOTNOTES - contimted
9 During coding, five codes of conversational moves were identified and categorized as not
relating specifically to any judgment-based or reasoning-based stopping rule. These expressions
included: ( a) statements in which the subject read a passage or phrase; (b) statements of a final
conclusion relative to the task of making a prediction ("I think housing sales will increase"); (c)
statements expressing affect ("I hate this bank stuff'); (d) questions and responses to the
interviewer, including statements of task strategy; and ( e) statements expressing a rescaling or
translation of previously accessed information, unrelated to drawing a specific conclusion.
Interviewer's comments were not coded.
29

Information
Manipulations in
3 Contrasting
Situations:
Stopping Rules
Magnitude
Threshold
Difference
Threshold
Representational
Stability
Propositional
Stability
TABLEt
Summary of Stopping Rule Predictions.
Familiar (F)
Ys.
New (N)
Alternating (A)
vs.
Unanimous
Support (U)
F>N A>U
Contradictory (CT)
Ys.
Continuing (CF)
CT > CF
F CF
FU CT = CF
Mental List F>N A-V
CT = CF
. Each cell in the table represents a partiCltlar stopping nile's prediction for that experimental
manipulation. The prediction serve.f as the criterion.for indicating the IlSe of the role.
30

TABLE 2
Experiment I
Frequencies of Paired Differences in Participants' Responses across Settings
S I
BF -BN

TABLE 3
Experiment 2
Frequencies of Paired Differences in Participants' Responses across Settings
32

TABLE 4
Experiment 3
Frequencies of Paired Differences in Participants' Responses across Settings
33

TABLES
Classification of Judgment Codes
34

TABLE 6
Classification of Mental Representation Codes
35

TABLE 7
Correspondence between Stopping Rules
and Judgment and Mental Representation Codes
tt Zero Group tt Responses Protocol Code
Difference Threshold I B...nc. WI, W...nc
Propositional Stability RC~usion
Mental List
"Greater-Than-Zero Group" of Re.\"pon.\"es
MSMt
Magnitude Threshold Bal,' WaU
Representational Stability RCI8Ik,RCextcmaI
36

IExperiment 1
Experiment 2
Experiment 3
TABLE 8
Stopping Rules Implicated by Results in Experiments .]
Magnitude Threshold
Mental List
Difference Threshold
Mental List I
TWO SUBGROUPS PROTOCOL
ANALYSIS
II Greater- Than-Zero II Group
Magnitude Threshold
Representational Stability
"Zero" Group
Difference Threshold
Propositional Stability
Mental List I
37
SUGGESTS
Magnitude Threshold
(Balance)
Propositional Stability
Mental List:

FIGURE 1
(0) Expcrimcnt 1 - P:lircd DiffcrcnceJ HF-HN, BF-BN
. ...
.
.
.
:.::::
...
...
:.
+ + + + + HF-HN
-3.0 0.0
mean - 0.867;
3.0 6.0
median - 1.000; stdev
9.0
- 2.270
12.0
. .
. . ..
. .. ..
. . .. . . . .
+ + + + + SF-aN
-3.0 0.0 3.0 6.0 9.0 12.0
mean - 1.900; median - 1.500; stdev ~ 4.551
(b) Expcrimcnt 2 - Paired Differences HA-HU, BA-BU
. .
.. .
.
-+
.
. :
+
..
.
:
.
:
+
.
:
.
.
: ...
+ + +
.
HA-HU
-6.0 . -3.0 mean - 0.500; 0.0 median - 3.0 0.000;
. . .
stdev 6.0 - 2.874 9.0
. . .
. . .
..
.
.
:
.
.
:
.
.
: : . 1 .
-+ + + + + + BA-BU
-6.0 -3.0 0.0
mean - 0.400,. median
3.0
- 0.000;
6.0
~tdev - 2.078
9.0
. .
. .
0.0 +
(c) Experiment 3 - Paircd Diffcrcnces HCT-HCF. BCT-BCF
. .. ..
+ + + + + HCT-HCF
0.4 1.5 3.0 4.5 6.0 7.5
_.n - 0.600; median. 0.000:. .stdev. 1.192
.
.
4
I.!
..n.
.00,. .
-+ +
3 .0
HlaD
4.$
- 0.001
38
.
+ BCT-BCF
0 7.5
. 1.408