Qualitative_data_analysis

258 QUALITATIVE DATA ANALYSIS
whether we are producing a three page summary for policy-makers or a three
hundred page thesis, there are some general criteria we have to address irrespective
of how we report the results of our analysis.
What criteria does an ‘acceptable’ account have to meet? We can employ the
three standard criteria for any analytic work. Is it reliable? Is it valid? And how
representative is it? These criteria are really quite simple, as we can see if we take the
example of telling the time. If my watch is reliable, it will be consistent, going
neither fast nor slow. If my watch is valid, it will tell the right time. If my watch is
representative, I’ll know that other people (in a particular population) share the
same time. An acceptable account has to convince its audience that it can meet each
of these criteria. Let us consider each in turn.
The essence of reliability is consistency through repetition. Suppose my watch is
wrong. It may be unreliable, or simply set at the wrong time (i.e. invalid). If I want
to know whether my watch is reliable, I need to make repeated observations of the
time. If I set it accurately and then, after an interval of say fifteen minutes, it is no
longer accurate, then I know it is unreliable. If it is accurate, can I infer that it is
reliable? In practice, I might —#8212;but not if my life depended on it! The
interval may be too short to show up error. Or it could be that my watch is very
erratic, sometimes going too fast and sometimes too slow, and by pure chance was
again telling the right time. If I can obtain consistent results over repeated
observations, at wider intervals, then this will give me more confidence that my
watch is reliable. Notice how much harder it is to be positive than negative. It may
take many repeated observations to acquire confidence in the watch’s reliability, but
only one negative observation to undermine it.
If our research is reliable, then others using the same procedures should be able to
produce the same result. The trouble arises because analytic procedures are typically
ill-defined, and replication by others is in any case a difficult if not impossible task.
I suggested earlier that in corroborating evidence, we have to undertake ‘internal
replication’ to test the reliability of our analysis (cf. Shimahara 1988). We may
obtain some assurance in this way that we at least can reproduce the same results by
using the same procedures on other parts of our data. But how do we assure others
of the reliability of our analysis?
Suppose I want to vouch for the reliability of my watch, but cannot let others use
it to make repeated measures. Not surprisingly, they are liable to become suspicious
of its reliability. How could I convince them otherwise? My only option is to
explain to them how the watch works, and convince them that every precaution has
been taken to ensure that it works as expected. In other words, I would have to
explain the principles of the measurement I am making, and what steps if any I have
taken to eliminate or reduce potential sources of error. Depending on how my
watch (or clock) operates, I may have to explain the mechanics of a pendulum or