01 NRDC Dyslexia 1-88 update - Texthelp

80
Research Report
devising. There might be assignment bias, which can occur when members of the intervention
group differ from controls in ways that predict a better response to the intervention, perhaps
by having higher motivation or greater ability. Although there are procedures to avoid these
kinds of bias, they are not always followed in intervention studies.
A fourth group of challenges is associated with shortcomings in the administration of the
intervention, such as insufficient treatment duration, differential withdrawal from the
programme on the part of successful and unsuccessful participants, or lack of follow-up to
determine whether the intervention has had a lasting effect. Inappropriate statistical
analyses, especially those which violate the assumptions of the tests used, are not
uncommon. These violations can suggest that differences in outcome are statistically
significant when they could have occurred by chance. Alternatively, differences in outcome
might be statistically significant but trivial to the policy-maker.
A further question can be asked of any apparently-effective intervention, namely: ‘Will it be
effective for others?’ This is an especially pertinent question to ask of the lateral visual
masking study of ‘four severe adult dyslexics’ by Geiger et al. (1992). Even if the initial
diagnosis was accurate (and if the intervention addressed the participants’ problem
successfully), no finding from such a small-scale study can be generalised with confidence to
a larger population. For future researchers, studies like this one can be important, but for
policy-makers they offer little guidance.
A different question can be asked of the study of thirty college students (Guyer, Banks, &
Guyer, 1993): ‘Is the superior intervention the most effective one available?’ One of the two
interventions in the comparison was an explicit, structured remedial spelling programme
using analytic and synthetic phonics. The comparison programme ‘taught spelling using a
non-phonetic approach’. In effect, the programmes were horses for different courses, one for
regularly-spelled words and one for irregular words. The outcome was that the ‘regular’ horse
ran faster than the ‘irregular’ horse, since most spellings are regular. Readers of a specialist
dyslexia journal might be disposed to believe that the ‘regular’ horse (in the form of an Orton-
Gillingham programme) would have won the gold cup in any comparison, but in this
comparison the race was fixed and the finding offers no answer to the question.
There are many pitfalls in evaluation research.
How can we know what really works?
What kinds of intervention study are likely to offer reliable guidance to tutors? This is not an
idle question. The design of intervention studies is exacting in any circumstances; it is
especially so when the participants are hard to reach. Perhaps for these reasons, there are
many design limitations in existing reading intervention research (Abadzi, 1994; Beder, 1999;
Lyon & Moats, 1997; Simmerman & Swanson, 2001; Torgerson et al., 2002).
Nevertheless, a good intervention study should satisfy a number of criteria. They include:
■
■
■
a focused research question, specifying a theoretically plausible mechanism of change;
a design that permits investigators to minimise the effects of extraneous variables, such as
teacher effects and differences in motivation;
a large enough sample to reduce the possibility that the outcomes of the intervention might
have occurred by chance;