Estimation, Evaluation, and Selection of Actuarial Models

Chapter 4
Model evaluation and selection
4.1 Introduction
When using data to build a model, the process must end with the announcement of a “winner.”
While qualifications, limitations, caveats, and other attempts to escape full responsibility are appropriate,
and often necessary, a commitment to a solution is required. In this Chapter we look at
a variety of ways to evaluate a model and compare competing models. But we must also remember
that whatever model we select, it is only an approximation of reality. This is reflected in the
following modeler’s motto: 1
“All models are wrong, but some models are useful.”
Thus, the goal of this Chapter is to determine a model that is good enough to use to answer the
question. The challenge here is that the definition of “good enough” will depend on the particular
application. Another important modeling point is that a solid understanding of the question will
guide you to the answer. The following quote from John Tukey sums this up:
“Far better an approximate answer to the right question, which is often vague, than an
exact answer to the wrong question, which can always be made precise.”
In this Chapter, a specific modeling strategy will be considered. For Part 4, our preference is to
have a single approach that can be used for any probabilistic modeling situation. A consequence is
that for any particular modeling situation there may be a better (more reliable or more accurate)
approach. For example, while maximum likelihood is a good estimation method for most settings,
it may not be the best 2 for certain distributions. A literature search will turn up methods that have
been optimized for specific distributions, but they will not be mentioned here. Similarly, many of
the hypothesis tests used here give approximate results. For specific cases, better approximations,
or maybe even exact results, are available. They will also be bypassed. The goal here is to outline
a method that will give reasonable answers most of the time and be adaptable to a variety of
situations.
1 It is usually attributed to George Box.
2 There are many definitions of “best.” Combining the Cramér-Rao lower bound with Theorem 3.29 indicates
that maximum likelihood estimators are asymptotically optimal using unbiasedness and minimum variance as the
definition of best.
61