Documentation of the Evaluation of CALPUFF and Other Long ...

C.4.2.1 SUMMARY OF LRT MODEL RANKINGS FOR CTEX3 USING STATISTICAL PERFORMANCE
MEASURES
Table C‐5 summarizes the rankings between the six LRT models for the 11 performance
statistics analyzed. Depending on the statistical metric, three different models were ranked as
the best performing model for a particular statistic with CAMx being ranked first more than the
other models (46%) and FLEXPART ranked first second most (36%). CALGRID was consistently
ranked the worst performing model being the poorest performing model for 6 of the 11
performance statistics.
In testing the efficacy of the RANK statistic for providing an overall ranking of model
performance we the ranking of the six LRT models using the average rank of the 11
performance statistics versus the ranking from the RANK statistical metric (Table C‐5). The
average rank of model performance for the six LRT dispersion models and the CTEX3
experiment averaged across all 11 performance statistics and the comparison to the RANK
rankings was as follows:
Ranking Average of 11
Statistics
RANK
1. CAMx CAMx
2. SCIPUFF SCIPUFF
3. FLEXPART FLEXPART
4. HYSPLIT CALPUFF
5. CALPUFF HYSPLIT
6. CALGRID CALGRID
For the CTEX3 experiment, the average rankings across the 11 statistics is nearly identical to the
rankings produced by the RANK integrated statistics that combines the four statistics for
correlation (PCC), bias (FB), spatial (FMS) and cumulative distribution (KS) with only HYSPLIT
and CALPUFF exchanging places as the 4 th and 5 th best performing models. CALPUFF
performance was weighted down in the average statistic rankings due to lower scores in the
FA2 and FA5 metrics compared to HYSPLIT. If not for this, the average rank across all 11 metrics
would have been the same as Draxler’s RANK score. Although this deviation did occur in the
fourth and fifth ranked positions, the RANK statistic remains a valid performance statistic for
indicating over all model performance of a LRT dispersion model. However, the analyst should
use discretion in relying too heavily upon RANK score without consideration to which
performance metrics are important measures for the particular evaluation goals. For example,
if performance goals are not concerned with a model’s ability to perform well in space and
time, then reliance upon spatial statistics such as the FMS in the composite RANK value may not
be appropriate. In the case of this evaluation, since space/time considerations are paramount
for proper LRT model performance, the RANK metric is a valuable tool to rapidly assess model
performance across a broad range of metrics being evaluated.
32