Downloadable - About University

Consistency and coherence checks 285
mean testing to see if an assessment obtained by different methods or
approaches is firm). Our view is that when a probability distribution
has to be assessed the probability method is usually the best approach to
adopt because, as we argued earlier, it tends to overcome the propensity
of decision makers to estimate distributions which have too narrow
arange.
Consistency and coherence checks
Consistency checks are, of course, a crucial element of probability
assessment. The use of different assessment methods will often reveal
inconsistencies that can then be fed back to the decision maker. These
inconsistencies should act as a stimulant to more intense thought which,
hopefully, will result in greater insight and improved judgment. Indeed,
the axioms of probability theory give no guidance as to which is the best method
for the elicitation of subjective probability. Empirical research in the psychological
laboratory has shown that sometimes the indirect methods
are inconsistent with direct methods and sometimes they are not. Some
investigators have demonstrated consistency between probability estimates
inferred from wagers and direct estimates. 8 Others have shown
that statistically naive subjects were inconsistent between direct and
indirect assessment methods, whereas statisticians were not. 9 Generally,
direct odds estimates, perhaps because they have no upper or lower
limit, tend to be more extreme than direct probability estimates. If probability
estimates derived by different methods for the same event are
inconsistent, which method should be taken as the true index of degree
of belief?
One way to answer this question is to use a single method of assessing
subjective probability that is most consistent with itself. In other words,
there should be high agreement between the subjective probabilities,
assessed at different times by a single assessor for the same event, given
that the assessor’s knowledge of the event is unchanged. Unfortunately,
there has been relatively little research on this important problem. One
review evaluated the results of several studies using direct estimation
methods. Test–retest correlations were all above 0.88 with the exception
of one study using students assessing odds – here the reliability was
0.66. It was concluded that most of the subjects in all experiments were
very consistent when using a single assessment method.
The implications of this research for decision analysis are not clear
cut. The decision analyst should be aware that different assessment