Documentation of the Evaluation of CALPUFF and Other Long ...

2009a). The key findings from the CTEX5 MM5 and CALMET meteorological evaluation are as
follows:
• The MM5 wind speed, and especially wind direction, model performance is better when
FDDA is used then when FDDA is not used.
• The “A” and “B” series of CALMET simulations produce wind fields least similar to the
MM5 simulation used as input, which is not surprising since CALMET by design is
modifying the winds at the location of the monitoring sites to better match the
observations.
• CALMET tends to slow down the MM5 wind speeds even when there are no wind
observations used as input (i.e., the “D” series).
• For this period and MM5 model configuration, the MM5 and CALMET wind model
performance is better when 12 km grid resolution is used compared to coarser resolution.
CAPTEX CALPUFF Model Evaluation and Sensitivity Tests
The CALPUFF model was evaluated against tracer observations from the CTEX3 and CTEX5 field
experiments using meteorological inputs from the various CALMET sensitivity tests described
above as well as the MMIF tool applied using the 80, 36 and 12 km MM5 databases. The
CALPUFF configuration was held fixed in all of these sensitivity tests so that the effects of the
meteorological inputs on the CALPUFF tracer model performance could be clearly assessed.
The CALPUFF default model options were assumed for most CALPUFF inputs. One exception
was for puff splitting where more aggressive vertical puff splitting was allowed to occur
throughout the day, rather than the default where vertical puff splitting is only allowed to occur
once per day.
The ATMES‐II statistical model evaluation approach was used to evaluate CALPUFF for the
CAPTEX field experiments. Twelve separate statistical performance metrics were used to
evaluate various aspects of the CALPUFF’s ability to reproduce the observed tracer
concentrations in the two CAPTEX experiments. Below we present the results of the RANK
performance statistic that is a composite statistic that represents four aspects of model
performance: correlation, bias, spatial and cumulative distribution. Our analysis of all twelve
ATMES‐II statistics has found that the RANK statistic usually provides a reasonable assessment
of the overall performance of dispersion models tracer test evaluations. However, we have
also found situations where the RANK statistic can provide misleading indications of the
performance of dispersion models and recommend that all model performance attributes be
examined to confirm that the RANK metric is providing a valid ranking of the dispersion model
performance.
CTEX3 CALPUFF Model Evaluation
Figure ES‐1 summarizes the RANK model performance statistics for the CALPUFF sensitivity
simulations that used the 12 km MM5 data as input. Using a 4 km CALMET grid resolution, the
EXP6B (RMAX1/RMAX2 = 100/200) has the lowest rank of the CALPUFF/CALMET sensitivity
tests. Of the CALPUFF sensitivity tests using the 12 km MM5 data as input, the CALPUFF/MMIF
(12KM_MMIF) sensitivity test has the highest RANK statistic (1.43) followed closely by EXP4A
(1.40; 12 km CALMET and 500/1000), EXP6C (1.38; 4 km CALMET and 10/500) with the lowest
13