RESEARCH METHOD COHEN ok

520 QUANTITATIVE DATA ANALYSIS
Box 24.15
Type I and Type II errors
Decision H 0 true H 0 false
Support H o Correct Type II error (β)
Do not support H 0 Type I error α Correct
One has to be careful not to describe an associative
hypothesis (e.g. gender) as a causal hypothesis, as
gender may not be actually having a causal effect.
In hypothesis testing one has to avoid Type I
and Type II errors. A Type I error occurs when one
does not support the null hypothesis when it is in
fact true. This is a particular problem as the sample
increases, as the chances of finding a significant
association increase, irrespective of whether a true
association exists (Rose and Sullivan 1993: 168),
requiring the researcher, therefore, to set a higher
alpha (α) limit (e.g. 0.01 or 0.001) for statistical
significance to be achieved). A Type II error occurs
when one supports the null hypothesis when it is
in fact not true (often the case if the levels of
significance are set too stringently, i.e. requiring
the researcher to lower the alpha (α) levelof
significance (e.g. 0.1 or 0.2) required). Type 1 and
Type II can be represented as in Box 24.15.
Effect size
One has to be cautious in using statistical
significance. Statistical significance is not the same
as educational significance. For example, I might
find a statistically significant correlation between
the amount of time spent on mathematics and
the amount of time spent in watching television.
This may be completely unimportant. Similarly I
might find that there is no statistically significant
difference between males and females in their
liking of physics. However, close inspection might
reveal that there is a difference. Say, for example,
that more males than females like physics, but that
the difference does not reach the ‘cut-off’ point of
the 0.05 level of significance; maybe it is 0.065. To
say that there is no difference, or simply to support
the null hypothesis here might be inadvisable.
There are two issues here: first, the cut-off level of
significance is comparatively arbitrary, although
high; second, one should not ignore coefficients
that fall below the conventional cut-off points.
This leads us into a discussion of effect size as an
alternative to significance levels.
Statistical significance on its own has come
to be seen as an unacceptable index of effect;
(Thompson 1994; 1996; 1998; 2001; 2002;
Fitz-Gibbon 1997: 43; Rozeboom 1997: 335;
Thompson and Snyder 1997; Wilkinson and the
Task Force on Statistical Inference, APA Board of
Scientific Affairs 1999; Olejnik and Algina 2000;
Capraro and Capraro 2002; Wright 2003; Kline
2004) because it depends on both sample size
and the coefficient (e.g. of correlation). Statistical
significance can be attained either by having a
large coefficient together with a small sample or
having a small coefficient together with a large
sample. The problem is that one is not able to
deduce which is the determining effect from a
study using statistical significance (Coe 2000: 9).
It is important to be able to tell whether it is the
sample size or the coefficient that is making the
difference. The effect size can do this.
What is required either to accompany or to
replace statistical significance is information about
effect size (American Psychological Association
1994: 18; 2001; Wilkinson and the Task Force
on Statistical Inference, APA Board of Scientific
Affairs 1999; Kline 2004). Indeed effect size is
seen as much more important than significance,
and many international journals either have
abandoned statistical significance reporting in
favour of effect size, or have insisted that statistical
significance be accompanied by indications of
effect size (Olejnik and Algina 2000; Capraro
and Capraro 2002; Thompson 2002). Statistical
significance is seen as arbitrary in its cut-off
points and unhelpful – a ‘corrupt form of the
scientific method’ (Carver 1978), an obstacle
rather than a facilitator in educational research.
It commands slavish adherence rather than
addressing the subtle, sensitive and helpful notion
of effect size (see Fitz-Gibbon 1997: 118). Indeed
commonsense should tell the researcher that a