Documentation of the Evaluation of CALPUFF and Other Long ...

discovered ratio based statistics such as FA2 and FA5 were highly susceptible to measurement
errors. Draxler proposed a single metric, which he calls RANK, which is the composite of one
statistical measure from each of the four broad categories.
( 1−
FB / 2 ) + FMS / 100+
( 1 KS / 100)
2
RANK R +
−
= (2‐13)
The final score, model rank (RANK), provides a combined measure to facilitate model
intercomparison. RANK is the sum of four of the statistical measures for scatter, bias, spatial
coverage, and the unpaired distribution. RANK scores range between 0.0 and 4.0 with 4.0
representing the best model ranking. Using this measure allows for direct intercomparison of
models across each of the four broader statistical categories.
2.4.3.4 Treatment of Zero Concentration Data
One issue in the performance evaluation was how to treat zero concentration data. Mosca et
al. (1998) filtered the ETEX observational dataset by only retaining non‐zero data and zero data
within two sample time intervals (6 hours) of the arrival and departure times of the tracer cloud
along with any zero observations in between these two time points. Stohl (1998) employed a
Monte Carlo approach by adding normally distributed “random errors” to the original values to
test the sensitivity of certain statistical measures to zero or near zero values. Stohl (1998)
identified that certain statistical parameters may be sensitive to small variations in
measurements when using “zero” or near “zero” background concentration data. While the
inclusion of “zero” data creates concern about the robustness of certain statistical measures,
especially ratio based statistics, there was also concern that only examining model statistics at
locations where the tracer cloud was observed provides a limited snapshot of a model’s
performance at those locations, and did not offer any insight into a model that may show
poorer performance by transporting emissions to incorrect locations or advection to correct
locations at incorrect times.
While the arguments for “filtering” of data are valid, it is also important to consider additional
statistical measures such as the FAR, POD, and TS where all zero data must be considered. All
zero data was retained for inclusion in the spatial analysis, but was filtered for the global
statistical analysis. The approach used in this project differs from the approach used by Draxler
et al. (2001) in that all zero‐zero pairs are considered in their analysis of HYSPLIT performance.
19