Documentation of the Evaluation of CALPUFF and Other Long ...

performing model the most often scoring best in 4 of the 11 statistics (36% of the time).
SCIPUFF, FLEXPART and CAMx all scored best with 2 of the 11 statistics (18%) with CALPUFF
scoring best for just one statistical metric.
In testing the efficacy of the RANK statistic, overall rank across all eleven statistics was used to
come up with an average modeled ranking to compare with the RANK statistic rankings. The
average rank across all 11 performance statistics and the RANK rankings are as follows:
Ranking Average of 11
Statistics
RANK
1. CAMx CAMx
2. HYSPLIT HYSPLIT
3. SCIPUFF CALGRID
4. FLEXPART SCIPUFF
5. CALPUFF FLEXPART
6. CALGRID CALPUFF
The results from CAPTEX Release 5 present an interesting case study on the use of the RANK
metric to characterize overall model performance. As noted in Table C‐6 and given above, the
relative ranking of models using the average rankings across the 11 statistical metrics is
considerably different than the RANK scores after the two highest ranked models. Both
approaches rank CAMx as the best and HYSPLIT as the next best performing models for CTEX5,
with rankings that are fairly close to each other. However, after that the two ranking
techniques come to different conclusions regarding the ability of the models to simulate the
observed tracer concentrations for the CTEX5 field experiment.
The most noticeable feature of the RANK metric for ranking models in CTEX5 is the third highest
ranking model using RANK, CALGRID (1.57). CALGRID ranks as the worst or second worst
performing model in 9 of the 11 performance statistics (82% of the time) and have an average
ranking of 5.0, which means on average it is the 5 th best performing model out of 6. In
examining the contribution to the RANK metric for CALGRID, there is not a consistent
contribution from all four broad categories to the composite score (Figure C‐40). Recall from
equation 2‐12 in Section 2.4.3.2 that the RANK score is defined by the contribution of the four
of the 11 statistics that represent measures of correlation/scatter (R2), bias (FB), spatial (FMS)
and cumulative distribution:
( 1−
FB / 2 ) + FMS / 100+
( 1 KS / 100)
2
RANK = R +
−
The majority of CALGRID’s 1.57 RANK score comes from fractional bias and Kolmogorov‐
Smirnov parameter values. Recall from Figures C‐36 and C‐39 that the FOEX and FB metrics
indicate that CALGRID consistently underestimates. The FB component to the composite score
for CALGRID is one of the highest among the six models in this study, yet the underlying
statistics indicate both marginal spatial skill and a degree of under‐prediction (likely due to the
spatial skill of the model).
The current form of the RANK score uses the absolute value of the fractional bias. This
approach weights underestimation equally to overestimation. However, in a regulatory
context, EPA is most concerned with models not being biased towards underestimation. When
looking at all of the performance statistics, CALGRID is clearly one of the worst performing LRT
41