Strategy Survival Guide

Strategy Survival Guide Version 2.1
Prime Minister’s Strategy Unit
home | strategy development | strategy skills | site index
Strategy Skills
> Building an Evidence Base
Looking forward - Counterfactual analysis
> in practice
Counterfactual analysis has theee broad aims:
• To establish evidence of a causal relationship between a new policy, or change in policy, and
outcomes the policy seeks to influence
• To account for confounding factors, additional to the influence of policy, that might lead to measured
change in outcomes
• To provide estimates of the impact of policy
What is it?
The counterfactual is an estimate of the circumstances that would have prevailed had a new policy or policy
change not been introduced. By comparing counterfactual outcomes (often referred to as either control or
comparison group outcomes) with outcomes measured for those units subject to the new policy or policy
change, causality or attribution can be established.
A counterfactual analysis tool used by government to identify causality or attribution is the use of pilots.
These enable the government to test new policies, or changes in existing policy, in a limited number of
geographical areas prior to introducing them more widely. The objective is to determine whether the new
policy gives rise to changes in the outcomes that policy seeks to alter. For example, counterfactual analysis
might answer the question – is there a direct relationship between a new initiative to cut car crime and
subsequent change in the number of reported car thefts, independent of other factor influencing car theft?
Counterfactual analysis explicitly acknowledges the fact that the outcomes government attempts to influence
are subject to a range of factors beyond the immediate scope of the policy being studied. For example, it
can’t necessarily be assumed that measures to cut worklessness are entirely responsible for an observed fall
in aggregate unemployment.
Units exposed to the new policy or policy change are alternatively referred to as the programme, treatment
or action group. In theory, causality can be attributed to the new policy because there are no systematic
differences between the programme group and a 'true’ counterfactual group, except for the fact that the
programme group has been exposed to the new policy. Differences in average outcomes between the
programme group and the 'true’ counterfactual group therefore represents an unbiased measure of the
programme’s impact.
In reality, measuring the counterfactual is a difficult task. Evaluators use a variety of methods, depending on
circumstances and opportunities open to them, to estimate the 'true’ counterfactual. The following
approaches can be used:
• Single group pre and post-test designs
• Two group pre and post-test designs
• Model-based econometric methods (simple regression adjustment, instrumental variables (IV), the
Heckman selection estimator)
• Statistical matching designs (e.g., propensity score or cell matching)
• Interrupted time series analysis
• Regression discontinuity designs
• Randomised control trial (RCT) designs (alternatively referred to as random assignment, random
allocation, experimental or randomised field trial designs).
It is the latter of these that is considered to be the most powerful method of establishing a net effect over and
above the counterfactual. This is because programme evaluators explicitly construct control and programme
Strategy Survival Guide – Strategy Skills
Page 165