Documentation of the Evaluation of CALPUFF and Other Long ...

The final panel in Figure C‐3 (bottom right) displays the overall RANK statistic. The RANK
statistics orders the model performance of the HYSPLIT INITD configurations as follows:
1. INITD1 (1.25)
2. INITD2 (1.21)
3. INITD104 (1.19)
4. INITD130 (1.19)
5. INITD0 (1.18)
6. INITD103 (1.15)
7. INITD4 (1.14)
8. INITD3 (1.11)
9. INITD140 (1.10)
The RANK performance statistics results presented above raise some interesting questions
about the RANK metric. The puff based configurations (INITD1 and INITD2) are the highest
ranking with scores using the RANK metric with values of 1.25 and 1.21 respectively. However,
each of these options had the worst (highest) NMSE and FB scores, while puff‐particle
configurations ranking slightly less using the RANK metric (1.1 to 1.19) have NMSE scores that
are much better (only one‐third) those for the puff configurations as well as slightly lower FB
scores. On the basis of RANK scores, the INITD1 and INITD2 configurations are the best
performing, but based upon other model performance statistics that are not included as the
four statistical metrics that make up the RANK metric (i.e., PCC, FB, FMS and KSP), the puff‐
particle hybrid configurations are better performing. Thus care must be taken in interpreting
model performance based solely on the RANK score and its use in performing model
intercomparisons and we recommend examining the whole suite of statistical performance
metrics, as well as graphical representation of model performance, to come to conclusions
regarding model performance.
C.2.3 HYSPLIT SPATIAL STATISTICS FOR CAPTEX RELEASE 5
Figure C‐4 displays the spatial model performance statistics for the HYSPLIT INITD sensitivity
tests for CAPTEX Release 5. Overall, the spatial performance for this experiment is very similar
to the results obtained from the ETEX INITD sensitivities for HYSPLIT. The puff configurations
(INITD1 and INITD2) exhibited the poorest performance across all of the spatial statistics.
INITD2 had the poorest FMS score with 5%, followed by INITD1 with 9.6%. INITD3 had the best
FMS score of 19.66%, but less than 2% separated all of the remaining particle and puff‐particle
INITD configurations. The particle mode (INITD0) exhibited the best TS with 24.4% with less
than 1.5% separating INITD103, 130, and 140 from INITD0. Consistently, the puff configurations
exhibited the lowest TS among the nine configurations, both with 7.9%.
5