Gregory West

Evaluation of Deterministic and
Probabilistic Meteorological
Forecasts for BC Watersheds
Greg West
Dominique Bourdin, Katelyn Wells,
Doug McCollor, and Roland Stull

Outline
• OBJECTIVE: Use wealth of verification data/tools to
evaluate meteorological models over BC watersheds
• Intro / Overview of Forecast and Verification System
and Website
• Deterministic Verification (2009‐2010)
• Bias
• Mean Absolute Error
• Z‐Score
• Mean Absolute Error Skill Score
• Summary of Preliminary Findings

End Users
• Meteorologists
• Hydrologists
• Load Forecasting
• Field Safety
• Planners
• Trading

Models
• Deterministic
– GEM
– GFS
• Probabilistic
– NAEFS
– UBC SREF
• ~100 pt fcsts
• Post‐processed
(McCollor and Stull 2008)
– Bias correction

Vancouver
Island
Bridge/
Lower
Mainland
Williston/
Peace
Columbia

NAEFS
Meteograms
• NAEFS‐PP
• CDF shaded
• GEM‐PP, GFS‐PP
• Cumulative Pcp
• Identify outliers
• User Concerns

Evaluation
Real‐time
•5‐Day, 30‐Day
•GEM/GFS/NAEFS‐PP
•Bias, MAE, Correlation

Evaluation
30‐Day Plots
•GEM/GFS‐PP
•History of forecast error

Evaluation
Water Yr Stats
•NAEFS‐PP, GEM‐DMO,
GEM‐PP, GFS‐PP

Bias
• Forecast –Observed Temperature (˚C)
• Forecast/Observed Precipitation (ratio)

GEM‐DMO Bias
• Tmin Coastal biases
generally +1‐3˚C
– Columbia +2‐4˚C
– Williston +3.5‐8˚C
• Tmax biases ‐1‐3˚C
– Bridge ‐3.4˚C
• Pcp biases generally positive
– Except higher elevation coastal
sites
• Raw model output does not
capture full evolution of
boundary layer/diurnal cycle.

GEM‐PP Bias
• Tmin/max Bias small
+/‐0.2˚C
• Tmin/max biases a bit
higher over Columbia,
within +/‐0.4˚C
• Pcp biases all‐region
average +2% (days1‐3),
+6‐9% (days 4‐6), and
~10‐20% (days 7‐8)
– +35‐50% for North Shore
Mtns
• Post‐processing very
effective at bias removal

GFS‐PP Bias
• Coastal and Bridge very
small biases
– BCK larger biases
– BRI ‐0.5˚C bias
• Columbia Tmin bias small,
but higher than Cst
• Columbia Pcp bias small but
bigger than Coastal, some
stns in particular
• Williston Pcp bias larger,
many +15‐30%, avg +13%

NAEFS‐PP Bias
• Most regions all biases very small thru day 16
• Coastal Pcp generally small pos bias ~+10% ‐ BCK larger biases
• Bridge +/‐10% Pcp ‐ BRI larger negative Tmin bias
• Columbia Tmin biases small but larger than others
– Pcp biases +10‐30%
• Williston Pcp largest positive biases, data issues

MAE
• MAE a function of model skill, observed
variability, and systematic bias
– Can’t compare stations/regions
• Model performance characteristics

GEM‐DMO MAE
• Coastal stns generally 2‐>3˚C,
– Tmax > Tmin errors
– high elevation stns Tmax 4‐5˚C
– Pcp 3‐>10mm
• Bridge, Tmin 3‐>4˚C , Tmax 4‐>5˚C
– BRI Tmin 5‐>6˚C
– Pcp 2‐>4mm
• Columbia/Williston
– Tmin/max 2‐>5˚C
– Pcp 1‐4mm
• Larger errors likely due to poor
terrain representation, high
variance
• Temperature not well forecast

GEM‐PP MAE
• Coastal and Bridge 1.5‐>3˚C
– Tmin/max MAE generally reduced
w.r.t. DMO, esp systematic errors
– Coastal Pcp 3‐>9mm, slightly
reduced w.r.t. DMO
– BCK Pcp MAE worse w.r.t. DMO
• Columbia Tmin/max improved
1‐1.5˚C, to 1.8‐>3.5˚C
– Pcp negligible change w.r.t. DMO
• Williston Tmin/max 2.5‐>4˚C
– Improved 1‐4˚C, still worst
– Shallow intense inversions
– Pcp small improvement
• Good reduction for Tmin/max, small to no improvement for Pcp

GFS‐PP MAE
• Coastal, Bridge, Columbia
– ~0.3˚C worse than GEM‐PP
– Pcp similar, better days 5‐8 in
Bridge, Columbia
• Williston GFS‐PP ~0.5˚C worse
than GEM‐PP
– Pcp equal
• Tmax MAE < Tmin MAE for
Interior
– Well‐mixed BL, Cloud/wind
variations
– Tmin MAE smaller for Cst

NAEFS‐PP MAE
• Coastal, Bridge Tmin/max 1.5‐>
2.5˚C by day 8
– ‐>3˚C by day 16
– Pcp similar
• Columbia Tmin/max MAE starts
high, stays constant or decreases to
day 6, then increases
• Williston 3‐>3.75˚C at day 8, 5˚C
at day 16
• Tmax better for Interior , Tmin
better at Coast
• Errors decrease more slowly or
level off for days 8‐16
– errors become random

Z‐Score
• Z‐Score: Forecast MAE normalized by the
standard deviation of observations
– Negatively oriented (less is better)

Coastal Regions Z‐Score
• Day 1 Tmin 0.43‐0.82, GEM‐PP best
• Day 1 Tmax 0.36‐0.73 (better), GEM‐PP best
• GFS‐PP trails GEM‐PP by ~0.13
• NAEFS‐PP better starting at day 5

Columbia Region Z‐Score
• Errors (actually) ~0.1 higher than coastal regions (Tmin: 0.57‐
.95, Tmax 0.43‐0.68)
• GEM‐PP more of a lead over others for Tmin (compared to
other regions), GFS‐PP Tmin 0.23 worse, Tmax is 0.03 worse
• NAEFS‐PP has lowest Z‐Score starting at day 6

Williston/Peace Z‐Score
• GEM‐PP best, day 1 Tmin range 0.53‐0.8. Day 1 Tmax 0.34‐
0.68.
• GFS‐PP is 0.27 worse for Tmin, 0.04 worse for Tmax
• NAEFS‐PP has better Z Score starting at day 7

All Regions Z‐Score
• GEM‐PP best (Tmin 0.49, Tmax 0.38)
• NAEFS‐PP best starting at day 5

Z‐Score Summary
• Tmin scores worse than Tmax
• Tmin errors lowest over Vancouver Island,
highest over Columbia
• Tmax errors similar across regions
• Model skill lower over Interior
• Model scores change slowly through forecast
lead times, suggesting results are fairly robust
• Performance of models relative to each other
fairly consistent across regions
– GEM‐PP best days 1‐4, NAEFS‐PP best for days 5‐16

Mean Absolute Error Skill Score
• Forecast MAE relative to Climo MAE (variance
built in)
• Positively oriented (higher is better)

Coastal Region MAESS
• Tmin Day 1 GEM‐PP ~twice
as skillful as GFS‐PP
• Tmin NAEFS‐PP/CMCENS‐PP
do best at days 4/5‐10
• Tmax Day 1 GEM‐PP ~1.4
time as skillful as GFS‐PP
• Tmax NAEFS‐PP/CMCENS‐PP
best at days 4/5‐9
• Pcp GEM‐PP slightly more
skillful than GFS‐PP
• Pcp NAEFS‐PP generally best
at days 4/5‐10
Day 1 Tmax
Day 5 Tmax
Day10
Tmax

Bridge Region MAESS
• Models more closely
clustered
• Tmin/max similar to other
regions
• GEM‐DMO Tmax skill
negative on day 1
• GEM‐PP Pcp lowest skill of
all models on day 1
• GEM‐DMO best for Pcp days
1‐6, then Climo
Day 1 4 7 Pcp

Columbia Region MAESS
• Tmin MAESS negative at day 1 for
some models
• Tmin lowest day 1 skill of any
region
• GEM‐PP and CMCENS‐PP display
small Tmin skill ~0.02 at day 5
• Day 1 Tmax MAESS 0.48, ~average
• GEM‐PP best for Tmin/max and
Pcp through day 5
• NAEFS‐PP Tmin/max skill through
day 8/9
• Day 1 Pcp ~average
• NAEFS‐PP Pcp skill out to day 12
Day 1 Tmin
Day 5 Tmin

Williston/Peace MAESS
• Tmin/max average
• GEM‐PP Tmin/max best
through day 7/8 (late)
• NAEFS‐PP Tmin/max skill to
day 9
• Pcp 0.1 below average
• Pcp closely clustered
• Very small Pcp skill out to
day 15 (!)
– 2008‐2009 many regions
PCP skill to 13‐15 days
Day 1 5 Pcp
Day 10
Pcp

MAESS Summary
– Tmin day 1 GEM‐PP twice as skillful as GFS‐PP
– Tmax day 1 GEM‐PP ~1.4 times more skillful than GFS‐PP
– NAEFS/CMCENS‐PP best at days 4‐10
• Summary
– GEM‐PP leads initially with few exceptions
– Some models/variables/regions have negative skill on day 1,
suggests serious forecasting deficiencies
– Models are clustered more closely in skill for Pcp
– Change over from GEM‐PP to NAEFS‐PP should occur ~day 5
– Skill to 8‐10 days, perhaps optimally blend with Climo
– Bridge Pcp skill half that of other regions, GEM‐DMO best
• Precipitation shadow of the Coast Range, and frequent but light
precipitation events

Preliminary Conclusions
• Some problem/suspect stations apparent
• MAE Characteristics:
– Diurnal cycle not fully captured
– Coastal MAE 1.5‐>3˚C, Interior 1.8‐>3.5˚C
• DMO is 0.5‐1.5˚C worse
Questions?
• Post‐processing
– very effective at removing Temp bias, reducing MAE
– Precipitation bias greatly reduced (still positive), MAE
negligibly improved
• Forecast skill lower over Interior
– need finer res models and/or more sophisticated PP
• GEM‐PP best through day 4
• NAEFS‐PP best days 5‐10
Greg.West@bchydro.com