Downloadable - About University

Assessment of the validity of probability forecasts 289
These conditions for learning calibration as a skill are perhaps most
obvious in professional weather forecasting. However, as Einhorn 16
has argued, most judgmental forecasts are made without the benefit
of such accurate feedback. Einhorn traces these difficulties to two
main factors. The first is a lack of search for and use of disconfirming
evidence, and the second is the use of unaided memory for coding,
storing, and retrieving outcome information. In addition, predictions
instigate actions to facilitate desired outcomes and indeed, outcome
feedback can be irrelevant for correcting poor judgment. Einhorn gives
the following example:
Imagine that you are a waiter in a busy restaurant and because you cannot
give good service to all the people at your station, you make a judgment
regarding which people will leave good or poor tips. You then
give good or bad service depending on your judgment. If the quality of
service, in itself, has an effect on the size of the tip, outcome feedback
will ‘confirm’ the predictions (‘they looked cheap and left no tip – just as
I thought’). The extent of such self-fulfilling prophecies is much greater
than we think and represents a considerable obstacle to learning from
outcome feedback.
It is clear that such ‘treatment effects’ where actions in the world can
determine subsequent outcomes may be more prevalent in some types
of forecasting situations than in others. The implications of this research
for decision analysis practice are not clear cut. Most forecasting tasks
can be seen to be unlike weather forecasting in that there are actions
that the forecaster can take to avoid or facilitate possible futures. As
we shall see in Chapter 12, decisions once made are often ‘made’ to
work. It follows that simply keeping a tally of your probability forecasts
and the subsequent occurrence or non-occurrence of events may not
be helpful in evaluating the calibration of your judgmental forecasting.
However, such a tally is more likely to reveal poor validity than simply
reflecting on the quality of your own judgment. This conclusion is
founded on what has come to be termed the ‘hindsight bias’. Briefly,
Fischhoff 17 has shown that remembered predictions of future events
‘move’ towards the event’s occurrence or non-occurrence. For example,
a real prediction of 0.8 that an event will happen tends to be inflated,
in recall, with the knowledge that the event did, in fact, occur. Such
a hindsight bias limits the possibility of improving calibration in the
light of experience and tends to artificially inflate our confidence in our
own judgmental abilities, resulting in what Fischhoff has termed the
‘I-knew-it-all-along’ effect.