RESEARCH METHOD COHEN ok

CONSTRUCTING A TEST 431
(e.g. to assign grades) is to lose the very purpose
of the criterion-referencing (Gipps 1994: 85). For
example, if a student is awarded a grade E for
spelling in English, and a grade A for imaginative
writing, this could be aggregated into a C grade as
an overall grade of the student’s English language
competence, but what does this C grade mean It
is meaningless, it has no frame of reference or clear
criteria, it loses the useful specificity of the A and
Egrades,itisacompromisethatactuallytellsus
nothing. Further, aggregating such grades assumes
equal levels of difficulty of all items.
Of course, raw scores are still open to
interpretation – which is a matter of judgement
rather than exactitude or precision (Wiliam
1996). For example, if a test is designed to
assess ‘mastery’ of a subject, then the researcher is
faced with the issue of deciding what constitutes
‘mastery’ – is it an absolute (i.e. very high score) or
are there gradations, and if the latter, then where
do these gradations fall For published tests the
scoring is standardized and already made clear, as
are the conversions of scores into, for example,
percentiles and grades.
Underpinning the discussion of scoring is the
need to make it unequivocally clear exactly what
the marking criteria are – what will and will
not score points. This requires a clarification of
whether there is a ‘checklist’ of features that must
be present in a student’s answer.
Clearly criterion-referenced tests will have
to declare their lowest boundary – a cut-off
point – below which the student has been deemed
to fail to meet the criteria. A compromise can be
seen in those criterion-referenced tests that award
different grades for different levels of performance
of the same task, necessitating the clarification
of different cut-off points in the examination. A
common example of this can be seen in the GCSE
examinations for secondary school pupils in the
United Kingdom, where students can achieve a
grade between A and F for a criterion-related
examination.
The determination of cut-off points has been addressed
by Nedelsky (1954), Angoff (1971), Ebel
(1979) and Linn (1993). Angoff (1971) suggests
amethodfordichotomouslyscoreditems.Here
judges are asked to identify the proportion of
minimally acceptable persons who would answer
each item correctly. The sum of these proportions
would then be taken to represent the minimally
acceptable score. An elaborated version of this
principle comes from Ebel (1979). Here a difficulty
by relevance matrix is constructed for all
the items. Difficulty might be assigned three levels
(e.g. easy, medium and hard) and relevance might
be assigned three levels (e.g. highly relevant, moderately
relevant, barely relevant). When each and
every test item has been assigned to the cells of
the matrix, the judges estimate the proportion of
items in each cell that minimally acceptable persons
would answer correctly, with the standard
for each judge being the weighted average of the
proportions in each cell (which are determined by
the number of items in each cell). In this method
judges have to consider two factors – relevance
and difficulty (unlike Angoff (1971), where only
difficulty featured). What characterizes these approaches
is the trust that they place in experts in
making judgements about levels (e.g. of difficulty,
or relevance, or proportions of successful achievement),
that is they are based on fallible human
subjectivity.
Ebel (1979) argues that one principle in
assignation of grades is that they should represent
equal intervals on the score scales. Reference is
made to median scores and standard deviations,
median scores because it is meaningless to
assume an absolute zero on scoring, and standard
deviations as the unit of convenient size for
inclusion of scores for each grade (see also Cohen
and Holliday 1996). One procedure is thus:
Calculate the median and standard deviation
of the scores.
Determine the lower score limits of the mark
intervals using the median and the standard
deviation as the unit of size for each grade.
However, the issue of cut-off scores is complicated
by the fact that they may vary according to
the different purposes and uses of scores (e.g.
for diagnosis, for certification, for selection, for
programme evaluation, as these purposes will affect
the number of cut-off points and grades, and the
Chapter 19