RESEARCH METHOD COHEN ok

CONSTRUCTING A TEST 425
true/false statements
open-ended questions where students are given
guidance on how much to write (e.g. 300 words,
a sentence, a paragraph)
closed questions.
These items can test recall, knowledge, comprehension,
application, analysis, synthesis and evaluation,
i.e. different orders of thinking. These take
their rationale from Bloom (1956) on hierarchies
of thinking – from low order (comprehension, application),
through middle order thinking (analysis,
synthesis) to higher order thinking (evaluation,
judgement, criticism). Clearly the selection of the
form of the test item will be based on the principle
of gaining the maximum amount of information in
the most economical way. This is evidenced in the
use of machine-scorable multiple choice completion
tests, where optical mark readers and scanners
can enter and process large-scale data rapidly.
In considering the contents of a test the test
writer must also consider the scale for some kinds
of test. The notion of a scale (a graded system of
classification) can be created in two main ways
(Howitt and Cramer 2005: 203):
Alistofitemswhosemeasurementsgofrom
the lowest to highest (e.g. an IQ test, a measure
of sexism, a measure of aggressiveness), such
that it is possible to judge where a student has
reached on the scale by seeing the maximum
level reached on the items;
The method of ‘summated scores’ (Howitt and
Cramer 2005: 203) in which a pool of items
is created, and the student’s score is the total
score gained by summing the marks for all the
items.
Further, many psychological tests used in
educational research will be unidimensional, that
is, the items all measure a single element or
dimension. Howitt and Cramer (2005: 204) liken
this to weighing 30 people using 10 bathroom
scales, in which one would expect a high
intercorrelation to be found between the bathroom
scales. Other tests may be multidimensional, i.e.
where two or more factors or dimensions are being
measured in the same test. Howitt and Cramer
(2005: 204) liken this to weighing 30 people
using 10 bathroom scales and then measuring
their heights using 5 different tape measures. Here
one would expect a high intercorrelation to be
found between the bathroom scale measures, a
high intercorrelation to be found between the
measurements from the tape measures, and a low
intercorrelation to be found between the bathroom
scale measures and the measurements from the tape
measures, because they are measuring different
things or dimensions.
Test constructors, then, need to be clear
whether they are using a unidimensional or a
multidimensional scale. Many texts, while advocating
the purity of using a unidimensional test
that measures a single construct or concept, also
recognize the efficacy, practicality and efficiency in
using multidimensional tests. For example, though
one might regard intelligence casually as a unidimensional
factor, in fact a stronger measure of
intelligence would be obtained by regarding it as
amultidimensionalconstruct,therebyrequiring
multidimensional scaling. Of course, some items
on a test are automatically unidimensional, for
example age, hours spent on homework.
Further, the selection of the items needs to be
considered in order to have the highest reliability.
Let us say that we have ten items that measure
students’ negative examination stress. Each item
is intended to measure stress, for example:
Item 1: Loss of sleep at examination time.
Item 2: Anxiety at examination time.
Item 3: Irritability at examination time.
Item 4: Depression at examination time.
Item 5: Tearfulness at examination time.
Item 6: Unwillingness to do household chores at
examination time.
Item 7: Mood swings at examination time.
Item 8: Increased consumption of coffee at
examination time.
Item 9: Positive attitude and cheerfulness at
examination time.
Item 10: Eager anticipation of the examination.
You run a reliability test (see Chapter 24 on SPSS
reliability) of internal consistency and find strong
intercorrelations between items 1–5 (e.g. around
Chapter 19