RESEARCH METHOD COHEN ok

424 TESTS
need to pilot home-grown tests. Items with limited
discriminability and limited difficulty must be
weeded out and replaced, those items with the
greatest discriminability and the most appropriate
degrees of difficulty can be retained; this can be
undertaken only once data from a pilot have been
analysed.
Item discriminability and item difficulty take on
differential significance in norm-referenced and
criterion-referenced tests. In a norm-referenced
test we wish to compare students with each other,
hence item discriminability is very important. In
acriterion-referencedtest,ontheotherhand,it
is not important per se to be able to compare
or discriminate between students’ performance.
For example, it may be the case that we wish
to discover whether a group of students has
learnt a particular body of knowledge, that
is the objective, rather than, say, finding out
how many have learned it better than others.
Hence it may be that a criterion-referenced test
has very low discriminability if all the students
achieve very well or achieve very poorly, but the
discriminability is less important than the fact
than the students have or have not learnt the
material. A norm-referenced test would regard
such a poorly discriminating item as unsuitable
for inclusion, whereas a criterion-referenced test
would regard such an item as providing useful
information (on success or failure).
With regard to item difficulty, in a criterionreferenced
test the level of difficulty is that
which is appropriate to the task or objective.
Hence if an objective is easily achieved then
the test item should be easily achieved; if the
objective is difficult then the test item should be
correspondingly difficult. This means that, unlike
anorm-referencedtestwhereanitemmightbe
reworked in order to increase its discriminability
index, this is less of an issue in criterionreferencing.
Of course, this is not to deny the
value of undertaking an item difficulty analysis,
rather it is to question the centrality of such a
concern. Gronlund and Linn (1990: 265) suggest
that where instruction has been effective the item
difficulty index of a criterion-referenced test will
be high.
In addressing the item discriminability, item
difficulty and distractor effect of particular test
items, it is advisable, of course, to pilot these tests
and to be cautious about placing too great a store
on indices of difficulty and discriminability that
are computed from small samples.
In constructing a test with item analysis, item
discriminability, item difficulty and distractor
effects in mind, it is important also to consider
the actual requirements of the test (Nuttall
1987; Cresswell and Houston 1991):
Are all the items in the test equally difficult
Which items are easy, moderately hard, hard
or very hard
What kinds of task is each item addressing:
is it a practice item (repeating known
knowledge), an application item (applying
known knowledge, or a synthesis item
(bringing together and integrating diverse areas
of knowledge)
If not, what makes some items more difficult
than the rest
Are the items sufficiently within the
experience of the students
How motivated will students be by the contents
of each item (i.e. how relevant will they
perceive the item to be, how interesting is it)
The contents of the test will also need to take
account of the notion of fitness for purpose, for
example in the types of test items. Here the
researcher will need to consider whether the kinds
of data to demonstrate ability, understanding and
achievement will be best demonstrated in, for
example (Lewis 1974; Cohen et al. 2004: ch. 16):
an open essay
afactualandheavilydirectedessay
short answer questions
divergent thinking items
completion items
multiple-choice items (with one correct answer
or more than one correct answer)
matching pairs of items or statements
inserting missing words
incomplete sentences or incomplete, unlabelled
diagrams