Measuring Impact - Nicva
Measuring Impact - Nicva
Measuring Impact - Nicva
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
58 <strong>Measuring</strong> <strong>Impact</strong> – Case-studies of impact assessment<br />
We tested a range of approaches in our four studies and, unsurprisingly, some worked better<br />
than others. Organisations will obviously need to think about the constraints in their own<br />
particular environment (resources, skills, access to information) when weighing up the pros<br />
and cons, but we have tried to assess the utility from the perspective of a small organisation<br />
with no dedicated research or evaluation function. Finally, this is not meant to be a critique of<br />
quantitative or qualitative methods, but a reflection on what might work in a voluntary and<br />
community-sector environment.<br />
8.3.1 Questionnaire-based approaches<br />
Using questionnaires to identify either perceptions of impact, or where and how your organisation<br />
has had an impact, is a relatively quick and cheap way of assessing impact. However,<br />
the problems may outweigh the benefits: respondents’ interpretations of questions vary significantly,<br />
while responses are obviously limited to the questions asked (so there is little scope<br />
for identifying unintended impact). Finally, for all of you suffering from consultation fatigue –<br />
do you really want to fill in another questionnaire<br />
8.3.2 Interview-based approaches<br />
Given the diversity and unpredictability of the themes outlined in Chapters 4 to 7, we believe<br />
that the assessment process must focus upon a qualitative appreciation of the impacts<br />
achieved. Interviews and focus groups worked particularly well for us, as this was a new and<br />
exploratory piece of work.<br />
With a loosely-structured, flexible interview schedule, each new interview can be used to<br />
inform subsequent ones. As a result the structure and themes evolve, leading to a broader<br />
view of impact. In effect, every interview is a pilot. This means that recorded interviews need<br />
to be transcribed and undergo basic analysis very soon after the event as new themes often<br />
emerge during the transcription and analysis stage.<br />
There was however an obvious drawback to this approach: the intensive use of time and<br />
resources. The time taken to undertake interviews was not adequately anticipated, and as a<br />
result two of the projects had to reduce the number of case-studies. To be useful this issue needs<br />
to be properly considered, especially as each case-study ideally needs interviews with each of the<br />
main stakeholder groups (paid staff, volunteers, trustees, users, partner organisations etc.).<br />
8.3.3 Use basic systems principles<br />
Another key learning point from the studies is the notion that impact is a social construct; that is,<br />
it varies according to where an individual or agency sits in relation to the activity under review.<br />
So, mapping out where stakeholders are in relation to that activity, and what their relationship is<br />
to that activity, is crucial in understanding how impact is both perceived and received.<br />
A formal ‘systems’ approach is not necessary to achieve this. However, the basic ideas used<br />
here can be incorporated into most types of impact measurement. Many approaches to<br />
impact assessment advise that you identify the intended beneficiaries and from this group<br />
select interviewees. The danger here is that you will fail to acknowledge any of the unanticipated<br />
impacts, and may introduce a positive bias. Allowing yourself to think about the<br />
system in which an organisation operates will reduce the likelihood of this happening and<br />
encourage you to think more broadly about possible impacts and impactees.