Documentation of the Evaluation of CALPUFF and Other Long ...

evaluation as the surface meteorological observations used in the evaluation were also used as
input into CALMET.
The METSTAT software (Emery et al., 2001) was used to match MM5 output with observation
data. The MMIFStat software (McNally, 2010) tool was used to match CALMET output with
observation data. Emery and co‐workers (2001) have developed a set of “benchmarks” for
comparing prognostic meteorological model performance statistics metrics. These benchmarks
were developed after examining the performance of the MM5 and RAMS prognostic
meteorological models for over 30 applications. The purpose of the benchmarks is not to
assign a passing or failing grade, rather it is to put the prognostic meteorological model
performance in context. The surface meteorological model performance benchmarks from
Emery et al., (2001) are displayed in Table 5‐7. Note that the wind speed RMSE benchmark was
also used for wind speed MNGE given the similarity of the RMSE and MNGE performance
statistics. These benchmarks are not applicable for diagnostic model evaluations.
Table 5‐7. Wind speed and wind direction benchmarks used to help judge the performance of
prognostic meteorological models (Source: Emery et al., 2001).
Wind Speed Root Mean Squared Error (RMSE)
≤ 2.0 m/s
Mean Normalized Bias (NMB)
≤ ±0.5 m/s
Index of Agreement (IOA)
≥ 0.6
Wind Direction Mean Normalized Gross Error (MNGE)
≤ 30°
Mean Normalized Bias (MNB)
≤ ±10°
Temperature Mean Normalized Gross Error (MNGE)
≤ 2.0 K
Mean Normalized Bias (NMB)
≤ ±0.5 m/s
Index of Agreement (IOA)
≥ 0.8
Humidity Mean Normalized Gross Error (MNGE)
≤ 2.0 g/kg
Mean Normalized Bias (NMB)
≤ ±1.0 g/kg
Index of Agreement (IOA)
≥ 0.6
The MM5 and CALMET comparisons to observations for CTEX3 and CTEX5 are provided in the
Appendix. The key findings of the CTEX5 MM5 and CALMET model performance evaluation are
as follows:
• The MM5 performance using the MRF PBL scheme (EXP2A‐C) was extremely poor. For
example the temperature exhibited an underestimation bias of over ‐4 °K, compared to
the benchmark of ≤±0.5 °K. Thus, MM5 sensitivity simulations using MRF PBL scheme
were discontinued.
• The MM5 wind speed, and especially wind direction, model performance is noticeably
better when FDDA was utilized.
• The “A” series of CALMET runs (RMAX1/RMAX2 = 500/1000) always has a wind speed
underestimation bias.
• The “C” and “D” series of CALMET sensitivity tests exhibit wind performance that is
comparable to the MM5 simulation used as input to CALMET.
• The 36 km and 12 km MM5 simulations exhibit substantially better model performance
than the 80 km MM5 simulation.
The CTEX3 and CTEX5 CALMET comparison for wind speed and direction needs to be viewed
with the caveat that because the winds are used as input in some of the sensitivity tests, then
this is not a true independent evaluation. Thus, it is at all not surprising that the CALMET wind
75