Measuring Impact - Nicva

58 Measuring Impact – Case-studies of impact assessment
We tested a range of approaches in our four studies and, unsurprisingly, some worked better
than others. Organisations will obviously need to think about the constraints in their own
particular environment (resources, skills, access to information) when weighing up the pros
and cons, but we have tried to assess the utility from the perspective of a small organisation
with no dedicated research or evaluation function. Finally, this is not meant to be a critique of
quantitative or qualitative methods, but a reflection on what might work in a voluntary and
community-sector environment.
8.3.1 Questionnaire-based approaches
Using questionnaires to identify either perceptions of impact, or where and how your organisation
has had an impact, is a relatively quick and cheap way of assessing impact. However,
the problems may outweigh the benefits: respondents’ interpretations of questions vary significantly,
while responses are obviously limited to the questions asked (so there is little scope
for identifying unintended impact). Finally, for all of you suffering from consultation fatigue –
do you really want to fill in another questionnaire
8.3.2 Interview-based approaches
Given the diversity and unpredictability of the themes outlined in Chapters 4 to 7, we believe
that the assessment process must focus upon a qualitative appreciation of the impacts
achieved. Interviews and focus groups worked particularly well for us, as this was a new and
exploratory piece of work.
With a loosely-structured, flexible interview schedule, each new interview can be used to
inform subsequent ones. As a result the structure and themes evolve, leading to a broader
view of impact. In effect, every interview is a pilot. This means that recorded interviews need
to be transcribed and undergo basic analysis very soon after the event as new themes often
emerge during the transcription and analysis stage.
There was however an obvious drawback to this approach: the intensive use of time and
resources. The time taken to undertake interviews was not adequately anticipated, and as a
result two of the projects had to reduce the number of case-studies. To be useful this issue needs
to be properly considered, especially as each case-study ideally needs interviews with each of the
main stakeholder groups (paid staff, volunteers, trustees, users, partner organisations etc.).
8.3.3 Use basic systems principles
Another key learning point from the studies is the notion that impact is a social construct; that is,
it varies according to where an individual or agency sits in relation to the activity under review.
So, mapping out where stakeholders are in relation to that activity, and what their relationship is
to that activity, is crucial in understanding how impact is both perceived and received.
A formal ‘systems’ approach is not necessary to achieve this. However, the basic ideas used
here can be incorporated into most types of impact measurement. Many approaches to
impact assessment advise that you identify the intended beneficiaries and from this group
select interviewees. The danger here is that you will fail to acknowledge any of the unanticipated
impacts, and may introduce a positive bias. Allowing yourself to think about the
system in which an organisation operates will reduce the likelihood of this happening and
encourage you to think more broadly about possible impacts and impactees.