Documentation of the Evaluation of CALPUFF and Other Long ...

Table ES‐6. Summary of model rankings using the statistical performance metrics and
comparison with the RANK metric.
Statistic 1 st 2 nd 3 rd 4 th 5 th 6 th
FMS SCIPUFF CAMx HYSPLIT CALPUFF FLEXPART CALGRID
FAR FLEXPART HYSPLIT CAMx SCIPUFF CALGRID CALPUFF
POD SCIPUFF CAMx HYSPLIT FLEXPART CALPUFF CALGRID
TS FLEXPART HYSPLIT CAMx SCIPUFF CALPUFF CALGRID
FOEX CALPUFF CAMx HYSPLIT CALGRID SCIPUFF FLEXPART
FA2 HYSPLIT CAMx CALPUFF SCIPUFF FLEXPART CALGRID
FA5 HYSPLIT CAMx SCIPUFF CALPUFF FLEXPART CALGRID
NMSE CAMx SCIPUFF FLEXPART HYSPLIT CALPUFF CALGRID
PCC or R HYSPLIT CAMx SCIPUFF FLEXPART CALGRID CALPUFF
FB CAMx CALGRID FLEXPART SCIPUFF HYSPLIT CALPUFF
KS HYSPLIT CALPUFF CALGRID CAMx FLEXPART SCIPUFF
Avg.
Ranking
CAMx HYSPLIT SCIPUFF FLEXPART CALPUFF CALGRID
Avg. Score 2.20 2.4 3.4 3.8 4.3 5.0
RANK
Ranking
CAMx HYSPLIT CALGRID SCIPUFF FLEXPART CALPUFF
RANK 1.91 1.80 1.57 1.53 1.45 1.28
European Tracer Experiment (ETEX)
The European Tracer Experiment (ETEX) was conducted in 1994 with two tracer releases from
northwest France that was measured at 168 samplers located in 17 European countries. Five
LRT dispersion models were evaluated for the first (October 23, 1994) ETEX tracer release
period (CALPUFF, SCICHEM, HYSPLIT, FLEXPART and CAMx). All five LRT dispersion models were
exercised using a common 36 km MM5 database for their meteorological inputs. For CALPUFF,
the MMIF tool was used to process the MM5 data. Default model options were mostly selected
for the LRT dispersion models. An exception to this is that for CALPUFF puff splitting was
allowed to occur throughout the day, instead of once per day which is the default setting. The
MM5 simulation was evaluated using surface meteorological variables. The MM5 performance
did not always meet the model performance benchmarks and exhibited a wind speed and
temperature underestimation bias. However, since all five LRT dispersion models used the
same MM5 fields, this did not detract from the LRT model performance intercomparison. The
ATMES‐II model evaluation approach was used in the evaluation that calculated 12 model
performance statistics of spatial, scatter, bias, correlation and cumulative distribution.
ETEX LRT Dispersion Model Performance Evaluation
Figure ES‐6 displays the ranking of the five LRT dispersion models using the RANK model
performance statistic with Table ES‐7 summarizing the rankings for the other 11 ATMES‐II
performance statistics. Depending on the statistical metric, three different models were ranked
as the best performing model for a particular statistic with CAMx being ranked first most of the
time (64%) and HYSPLIT ranked first second most (27%). In order to come up with an overall
rank across all eleven statistics we average the modeled ranking order to come up with an
average ranking that listed CAMx first, HYSPLIT second, SCIPUFF third, FLEXPART fourth and
CALPUFF the fifth. This is the same ranking as produced by the RANK integrated statistics that
combines the four statistics for correlation (PCC), bias (FB), spatial (FMS) and cumulative
distribution (KS), giving credence that the RANK statistic is a potentially useful performance
23