RESEARCH METHOD COHEN ok

324 QUESTIONNAIRES
the researcher may be asking which post-16 course
a student is following (e.g. academic, vocational,
manual, non-manual). In these cases only one
response may be selected. As with the dichotomous
variable, the listing of several categories or
elements of a variable (e.g. form membership
and course followed) enables nominal data to
be collected and processed using the chi-square
statistic, the G-test and cross-tabulations (Cohen
and Holliday 1996). Multiple choice questions are
treated as nominal data (see Part Five).
It may be important to include in the multiple
choices those that will enable respondents to
select the response that most closely represents
their view, hence a pilot is needed to ensure that
the categories are comprehensive, exhaustive and
representative. On the other hand, the researcher
may be interested in certain features only, and it is
these that would figure in the response categories
only.
The multiple choice questionnaire seldom gives
more than a crude statistic, for words are inherently
ambiguous. In the example above of chemistry, the
notion of ‘useful’ is unclear, as are ‘appropriate’,
‘practicable’ and ‘burden’. Respondents could
interpret these words differently in their own
contexts, thereby rendering the data ambiguous.
One respondent might see the utility of the
chemistry scheme in one area and thereby say
that it is useful – ticking (d). Another respondent
might see the same utility in that same one area
but, because it is only useful in that single area,
may see this as a flaw and therefore not tick
category (d). With an anonymous questionnaire
this difference would be impossible to detect.
This is the heart of the problem of questionnaires
– that different respondents interpret the
same words differently. ‘Anchor statements’ can
be provided to allow a degree of discrimination
in response (e.g. ‘strongly agree’, ‘agree’ etc.) but
there is no guarantee that respondents will always
interpret them in the way that is intended. In
the example above this might not be a problem
as the researcher might only be seeking an index
of utility – without wishing to know the areas of
utility or the reasons for that utility. The evaluator
might be wishing only for a crude statistic (which
might be very useful statistically in making a decisive
judgement about a programme). In this case
this rough and ready statistic might be perfectly
acceptable.
One can see in the example of chemistry
above not only ambiguity in the wording but
also a very incomplete set of response categories
which is hardly capable of representing all aspects
of the chemistry scheme. That this might be
politically expedient cannot be overlooked, for
if the choice of responses is limited, then those
responses might enable bias to be built into the
research. For example, if the responses were limited
to statements about the utility of the chemistry
scheme, then the evaluator would have little
difficulty in establishing that the scheme was
useful. By avoiding the inclusion of negative
statements or the opportunity to record a negative
response the research will surely be biased. The
issue of the wording of questions has been discussed
earlier.
Multiple choice items are also prone to
problems of word order and statement order. For
example, Dillman et al.(2003:6)reportastudyof
German students who were asked to compare their
high school teachers in terms of whether male
or female teachers were more empathetic. They
found that respondents rated their female teachers
more highly than their male teachers when asked
to compare female teachers to male teachers than
when they were asked to compare their male
teachers to their female teachers. Similarly they
report a study in which tennis was found to be less
exciting than football when the tennis option was
presented before the football option, and more
exciting when the football option was placed
before the tennis option. These studies suggest
that respondents tend to judge later items in terms
of the earlier items, rather than vice versa and
that they overlook features specific to later items
if these are not contained in the earlier items.
This is an instance of the ‘primacy effect’ or ‘order
effect’, wherein items earlier in a list are given
greater weight than items lower in the list. Order
effects are resilient to efforts to minimize them, and
primacy effects are particularly strong in Internet
questionnaires (Dillman et al. 2003: 22).