Downloadable - About University

288 Methods for eliciting probabilities
assessed probability is equivalent to proportion correct over a number
of assessments of equal probability. For example, if you assign
a probability of 0.7 as the likelihood of each of 20 events occurring,
14 of those 20 events should occur if your probability forecasts are
perfectly calibrated.
Earlier we discussed the usefulness of checking the consistency and
coherence of probability assessments. Perfect test–retest consistency is
a necessary but not sufficient condition for perfect coherence, and
perfect coherence in turn is a necessary but not sufficient condition
for perfect calibration. Consider the case of a sub-standard rule that
has been poorly manufactured such that when it reports a measure of
one meter it is in fact undermeasuring by one centimeter. In this case,
measurements by the rule would be consistent from one measurement
to another and would also be coherent in that a two-meter measurement
would additively consist of two one-meter measurements. However, the
measurement itself is, as we know, invalid. It follows that consistency
and coherence are necessary but not sufficient conditions for validity.
If assessed subjective probabilities are inconsistent and incoherent they
cannot be valid, i.e. well calibrated. The stages in the assessment of
probability forecasts are given in Figure 10.3.
Given that assessed probabilities are consistent and coherent then
validity becomes an issue. However, calibration is a long-run measure.
You cannot sensibly compute the calibration of a single probability
assessment except for that of 1.0, i.e. certainty. Although perfect
calibration is the most desirable aspect of judgmental probability
forecasts, in most practical circumstances it may not be possible to measure
this aspect of validity. Accordingly, attention should be focused
on the other indices of a probability assessment’s adequacy: consistency
and coherence. As we saw in Chapter 9, our view is that
performance-demonstrated expertise in judgmental probability forecasting
is underpinned by practice and regular performance feedback.
Consistency
Coherence
Calibration
Stage 1
Stage 2
Stage 3
Figure 10.3 – Stages in the assessment of probability forecasts