manual for social impact assessment of land-based ... - Forest Trends

• People in control groups are known to change their behavior when studied (known as the
‘Hawthorne effect’);
• The ethical problem of denying people in control groups the future right to participate in an
expanding project;
• Measuring impacts at one point of time is considered less reliable for establishing causality
than a comparison at two points of time;
• There is low motivation for control groups to cooperate.
In sum, the experimental approach is expensive and does not ensure attribution, due to possible
differences in unobservable characteristics or other causes of bias (Tanburn, 2008).
Quasi-experimental methods
The above problems usually mean that a pure statistical approach is not practical or viable for
impact assessment. Therefore a ‘quasi-experimental’ approach involving 'constructed controls' is
often used. This means trying to find people or groups who are as similar as possible to the project
participants in terms of their observable characteristics, possibly living in the same or in nearby
communities. But the problems of selecting controls are more or less the same, and it has a lower
level of certainty as regards attribution compared to the experimental method.
The quasi-experimental approach can theoretically be used with or without a starting conditions or
‘baseline’ study, but it is more reliable with a baseline – sometimes called the before-and-aftercontrol-impact
(BACI) design. Assuming a baseline is used, it is necessary to ‘over-sample’ by 50%
according to some sources – and possibly a higher percentage among the controls if there are few
incentives for them to remain in their communities. This allows for the natural decline or ‘attrition’
(e.g. outmigration, death, etc.) of respondents.
A cheaper but less reliable alternative to the above approaches is a 'before and after' comparison by
project participants, known as 'reflexive comparison'. In this case the ‘control’ element is provided
by the original conditions or before project situation. Again it is more effective and reliable with a
baseline. If used without a baseline, it depends entirely on memory recall, and is therefore more
subjective and open to bias. It is again important to 'over-sample' if a baseline is undertaken.
Data collection methods
As well as appropriate indicator selection, ‘matching methods’ need to be accompanied by carefully
chosen and designed data collection methods which are described in the subsequent sections of the
Toolbox. Thus, for example, the Basic Necessities Survey (BNS) is a cost-effective approach to
measuring changes in poverty in control and project communities (Section T6.1), while several of the
Participatory Impact Assessment (PIA) methods are relevant for matching methods (Section T6.2).
Social Impact Assessment of Land-Based Carbon Projects (1.0) – Part II | 33