Documentation of the Evaluation of CALPUFF and Other Long ...

The ETEX real‐time LRT modeling phase was performed in parallel with the tracer field
experiment. When the release started, 28 modeling groups were notified of the starting time,
source location, and emission rate. They ran their LRT models in real‐time to predict the
evolution of the tracer cloud, and their predictions were sent as soon as they were available to
the statistical evaluation team at JRC‐Ispra. The capability of providing these predictions in
real‐time was considered to be an important factor, as well as the model performance itself.
Therefore, only those institutions that had access to a meteorological model or that received
real‐time forecasts from a meteorological centre could participate.
The analysis of these calculations could not distinguish the differences between predictions and
measurements arising from dispersion model inadequacies as opposed to those arising from
the meteorological forecasts used. Almost two years after the ETEX releases, the ATMES‐II
modeling exercise was launched to evaluate the LRT models in hindcast mode. ATMES‐II
participants were required to calculate the concentration fields of the first ETEX tracer
experiment using ECMWF analyzed meteorological data as input to their own dispersion
models. Any institution operating a long‐range dispersion model could now participate
whether or not it had real‐time access to the meteorological data, and the number of
participants (49) was increased compared to the ETEX real‐time modeling exercise, even though
not all of the original ETEX modelers took part in ATMES‐II.
Contrary to ETEX, the differences between the measured and modeled concentration fields in
ATMES‐II could be more directly related to the dispersion simulation, thanks to the use of the
same meteorological fields. However, even in this case, discrepancies between models were
due not only to the calculation of dispersion, but also to the different ways in which the
meteorological information was used. Moreover, ATMES‐II modelers could also submit results
obtained with a meteorological analysis different from that of ECMWF.
As for the statistical analysis in ETEX real‐time modeling exercise, the analysis of ATMES‐II
model results was divided into time, space and global analyses. The same statistical indices of
the first ETEX release were computed in the time analysis, while for the other two analyses
some different indices were computed following the requirements of modelers, and the
experience gained during the two real‐time exercises.
In a general, a substantial improvement in the models' performance in the ATMES‐II modeling
was seen compared to the ETEX real‐time modeling phase for the common statistical indices.
When comparing the results of the ATMES‐II statistical analysis with those for the real‐time
simulation of the first ETEX release, a general improvement of the model performances for
those who took part in both exercises is evident. This can be explained by the better resolution
of the meteorological fields used, the availability of the measured values of tracer
concentration that allowed participants to tune some parameters in their long‐range dispersion
model and the time elapsed between the two exercises (2 years) during which improvements in
model formulation and application procedures took place.
Spatial Analysis: In ATMES‐II the spatial analysis consisted of the calculation of the Figure of
Merit in Space (FMS) at 12, 24, 36, 48, 60 hours after the release start. The FMS is the ratio of
the spatial distribution of the overlap of the predicted and observed tracer pattern to the union
of the predicted and observed tracer pattern and is expressed as a percent (note that all
statistical metrics are defined in detail in Section 2.4). A big improvement could be observed in
the models' FMS compared to the ETEX real‐time exercise for the first release. For instance, at
36 hours in ATMES‐II all the models had a non‐zero FMS, half of the models had FMS>45% and
10