RESEARCH METHOD COHEN ok

156 VALIDITY AND RELIABILITY
regression to the mean. Regression effects can
lead the educational researcher mistakenly
to attribute post-test gains and losses to low
scoring and high scoring respectively.
Testing: Pretests at the beginning of experiments
can produce effects other than those due
to the experimental treatments. Such effects
can include sensitizing subjects to the true
purposes of the experiment and practice effects
which produce higher scores on post-test
measures.
Instrumentation: Unreliable tests or instruments
can introduce serious errors into experiments.
With human observers or judges
or changes in instrumentation and calibration,
error can result from changes in their skills and
levels of concentration over the course of the
experiment.
Selection: Bias may be introduced as a result
of differences in the selection of subjects
for the comparison groups or when intact
classes are employed as experimental or control
groups. Selection bias, moreover, may interact
with other factors (history, maturation, etc.)
to cloud even further the effects of the
comparative treatments.
Experimental mortality: The loss of subjects
through dropout often occurs in long-running
experiments and may result in confounding
the effects of the experimental variables, for
whereas initially the groups may have been
randomly selected, the residue that stays the
course is likely to be different from the unbiased
sample that began it.
Instrument reactivity: The effects that the
instruments of the study exert on the people in
the study (see also Vulliamy et al.1990).
Selection-maturation interaction: This can occur
where there is a confusion between the research
design effects and the variable’s effects.
Threats to external validity
Threats to external validity are likely to limit
the degree to which generalizations can be made
from the particular experimental conditions to
other populations or settings. We summarize here
a number of factors (adapted from Campbell and
Stanley 1963; Bracht and Glass 1968; Hammersley
and Atkinson 1983; Vulliamy 1990; Lewis-Beck
1993) that jeopardize external validity.
Failure to describe independent variables explicitly:
Unless independent variables are adequately
described by the researcher, future replications
of the experimental conditions are virtually
impossible.
Lack of representativeness of available and
target populations: While those participating
in the experiment may be representative of
an available population, they may not be
representative of the population to which the
experimenter seeks to generalize the findings,
i.e. poor sampling and/or randomization.
Hawthorne effect: Medicalresearchhaslong
recognized the psychological effects that
arise out of mere participation in drug
experiments, and placebos and doubleblind
designs are commonly employed to
counteract the biasing effects of participation.
Similarly, so-called Hawthorne effects threaten
to contaminate experimental treatments in
educational research when subjects realize their
role as guinea pigs.
Inadequate operationalizing of dependent variables:
Dependent variables that experimenters
operationalize must have validity in the nonexperimental
setting to which they wish to
generalize their findings. A paper and pencil
questionnaire on career choice, for example,
may have little validity in respect of the actual
employment decisions made by undergraduates
on leaving university.
Sensitization/reactivity to experimental conditions:
As with threats to internal validity, pretests
may cause changes in the subjects’ sensitivity
to the experimental variables and thus cloud
the true effects of the experimental treatment.
Interaction effects of extraneous factors and
experimental treatments:Alloftheabovethreats
to external validity represent interactions of
various clouding factors with treatments. As
well as these, interaction effects may also
arise as a result of any or all of those factors