Documentation of the Evaluation of CALPUFF and Other Long ...

observations) divided by the number of monitor‐time events that either a prediction or
observed tracer occurred at a monitor (i.e., either a predicted or observed hits), with a perfect
score of 100% (which means there were no occurrences when there was a predicted hit but an
observed miss and vice versa).
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Threat Score (TS)
(Perfect = 100%)
CALPUFF SCIPUFF HYSPLIT FLEXPART CAMx
Figure 6‐8. Threat Score (TS) statistical performance metric for the five LRT models and the
ETEX tracer field experiment.
6.4.1.2 Global Analysis of Model Performance
Eight global statistical analysis metrics are used to evaluate the five LRT model performance
using the ETEX data base that are described in Section 2.4 and consist of the FOEX, FA2, FA5,
NMSE, PCC, FB, KS and RANK statistical metrics.
The Factor of Exceedance (FOEX) gives a measure of the scatter of the modeled predicted and
observed and a level of underestimation versus overestimation of the model. FOEX is bounded
by ‐50% to +50%. The within a Factor of α (FAα), where we used within a Factor of 2 (FA2) and
5 (FA5), also gives an indication of the amount of scatter in the predicted and observed tracer
pairs, but no information on whether the model is over‐ or under‐predicting. A perfect model
would have an FAα score of 100%. A good performing model would have a FOEX score near
zero and high FAα values. A model with a large negative FOEX and low FAα values would
indicate an under‐prediction tendency. Whereas a model with a large positive FOEX and low
FAα would suggest a model that over‐predicts.
Figure 6‐9 displays the FOEX performance metrics for the five LRT models and the ETEX
modeling period.
113