Documentation of the Evaluation of CALPUFF and Other Long ...

Documentation of the Evaluation of CALPUFF and Other Long ... Documentation of the Evaluation of CALPUFF and Other Long ...

20.04.2013 Views

6.4.1.1 Spatial Analysis of Model Performance Four spatial analysis model performance statics have been identified and are discussed in this section: FMS, FAR, POD and TS. Figure 6‐5 displays the FMS spatial analysis performance metrics for the five LRT models and the ETEX tracer study field experiment. Recall that the FMS statistic is define as the overlap divided by the union of the predicted and observed tracer clouds with a perfect model receiving an FMS score of 100%. 60% 50% 40% 30% 20% 10% 0% Figure of Metric in Space (FMS) (Perfect = 100%) CALPUFF SCIPUFF HYSPLIT FLEXPART CAMx Figure 6‐5. Figure of Merit in Space (FMS) statistical performance metric for the five LRT models and the ETEX tracer field experiment. Figure 6‐6 displays the False Alarm Rate (FAR) performance metrics. The FAR metric is defined by the number of times that a tracer concentration was predicted to occur at a monitor‐time when no tracer was observed (i.e., a miss) divided by the number of times a tracer was predicted to occur at a monitor‐time (i.e., sum of misses and hits); a perfect model (i.e., one that had no misses) would have a FAR score of 0%. 111

70% 60% 50% 40% 30% 20% 10% 0% Figure 6‐6. False Alarm Rate (FAR) statistical performance metric for the five LRT models and the ETEX tracer field experiment. The Probability of Detection (POD) performance statistic is defined as the percent of the time the predicted and observed tracer both occurred at a monitor‐time (i.e., a hit of tracer concentrations greater than 1 ngm ‐3 ) divided by the number of times that the tracer was observed at any monitor‐time (i.e., sum of hits and misses); a perfect model POD score would be 100% (i.e., anytime there was observed tracer at a monitor there was also predicted tracer at the monitor). 70% 60% 50% 40% 30% 20% 10% 0% False Alarm Rate (FAR) (Perfect = 0%) CALPUFF SCIPUFF HYSPLIT FLEXPART CAMx Probability of Detection (POD) (Perfect = 100%) CALPUFF SCIPUFF HYSPLIT FLEXPART CAMx Figure 6‐7. Probability of Detection (POD) statistical performance metric for the five LRT models and the ETEX tracer field experiment. The Threat Score (TS) is the ratio of the number of times that a tracer is both predicted and observed at a monitor‐time at the same time (i.e., common hits among the predictions and 112

70%<br />

60%<br />

50%<br />

40%<br />

30%<br />

20%<br />

10%<br />

0%<br />

Figure 6‐6. False Alarm Rate (FAR) statistical performance metric for <strong>the</strong> five LRT models <strong>and</strong><br />

<strong>the</strong> ETEX tracer field experiment.<br />

The Probability <strong>of</strong> Detection (POD) performance statistic is defined as <strong>the</strong> percent <strong>of</strong> <strong>the</strong> time<br />

<strong>the</strong> predicted <strong>and</strong> observed tracer both occurred at a monitor‐time (i.e., a hit <strong>of</strong> tracer<br />

concentrations greater than 1 ngm ‐3 ) divided by <strong>the</strong> number <strong>of</strong> times that <strong>the</strong> tracer was<br />

observed at any monitor‐time (i.e., sum <strong>of</strong> hits <strong>and</strong> misses); a perfect model POD score would<br />

be 100% (i.e., anytime <strong>the</strong>re was observed tracer at a monitor <strong>the</strong>re was also predicted tracer<br />

at <strong>the</strong> monitor).<br />

70%<br />

60%<br />

50%<br />

40%<br />

30%<br />

20%<br />

10%<br />

0%<br />

False Alarm Rate (FAR)<br />

(Perfect = 0%)<br />

<strong>CALPUFF</strong> SCIPUFF HYSPLIT FLEXPART CAMx<br />

Probability <strong>of</strong> Detection (POD)<br />

(Perfect = 100%)<br />

<strong>CALPUFF</strong> SCIPUFF HYSPLIT FLEXPART CAMx<br />

Figure 6‐7. Probability <strong>of</strong> Detection (POD) statistical performance metric for <strong>the</strong> five LRT<br />

models <strong>and</strong> <strong>the</strong> ETEX tracer field experiment.<br />

The Threat Score (TS) is <strong>the</strong> ratio <strong>of</strong> <strong>the</strong> number <strong>of</strong> times that a tracer is both predicted <strong>and</strong><br />

observed at a monitor‐time at <strong>the</strong> same time (i.e., common hits among <strong>the</strong> predictions <strong>and</strong><br />

112

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!