Measuring Impact - Nicva

Assessing the impact of regeneration projects 35
Critique of methodology 5.4
A number of lessons were learnt as a result of this approach for identifying different impacts.
Some concerned the use of a qualitative approach, whereas some were specific to the regeneration
partnership.
5.4.1 Qualitative approaches
In a study where interviews are being used to establish the range of impacts resulting from
particular activities, it is sometimes difficult to know to what extent the full range of possible
impacts has been successfully captured. Similarly, we will never know whether the full range
of stakeholders has been identified.
Another problem with this type of approach is that it is not always possible to know whether
interviewees are simply quoting from their organisation’s plans or objectives, and giving the
‘accepted line’ as regards particular types of impact. However, these issues can be addressed
by careful use of interview techniques.
5.4.2 Issues of unanticipated,collective and long-term impact assessment
Interviewees admitted that they don’t actually know what the impact of their work is. This
perhaps explains why, when asked about impact, people often talk about intended impact and
what they hope to achieve. There is an admission too, that knowledge of a project’s impact
often comes from hearsay, the sources of which may not be reliable. Sometimes the very nature
of a project means that some elements of its impact will not be directly visible to those involved
in delivering the project. Another problem is that assessing impact is particularly difficult when
the activity’s aim is to slow down a negative effect or prevent something from happening.
It was hoped that this approach would prove a successful means of capturing unanticipated
(including negative), collective and long-term impacts. As a whole the approach worked well,
but there were some limitations, particularly with regard to long-term and collective impact.
People found it hard to talk about the possible future impacts of a particular activity or
organisation. There was a general tendency, when talking about the future, to say things like
“well, what we hope to achieve is…” and “what our plan states is…”. In the absence of any
anecdotal evidence to illustrate the long-term or future impact of a project, people tend to fall
back on quoting the planned impacts, or objectives.
Identifying collective impact also proved difficult. An assumption was made that the projects
would consider themselves to be part of a larger whole or entity and share common aims. It
seemed however, that while projects certainly shared some common objectives and aims, most
objectives were organisation-specific. As a result, establishing collective impact in terms of
common aims was not possible. It was more relevant to explore the existence of collective
impact generated by the chosen projects and other funders or partners. These included development
agencies and other partnerships. Here, relationships and shared objectives tended to
be strong, and project managers had a clear sense of how they worked together to achieve
things that would not otherwise have been achieved.