Measuring Impact - Nicva

52 Measuring Impact – Case-studies of impact assessment
Critique of methodology 7.4
The study has explored the use of elements of a systems approach for developing indicators
to capture an organisation’s impact on social and human capital. This section will summarise
what has been learnt from this approach.
7.4.1 Systems approach
The use of a systems approach for developing a network map provided a structure on which
to base the interview plan. It provided a structure for dividing interviews into categories of
stakeholders. It also helped to identify groups working with AFG. This is useful for any
organisation as it shows outsiders the range of stakeholders that the group is working with.
7.4.2 Use of qualitative methods
Until fairly recently, organisations have tended to measure direct outputs such as numbers of
courses run and numbers of attendees on these courses. This study did use quantitative data
but went further by using interviews and focus groups to provide in-depth knowledge from
different perspectives, of the effects of projects carried out by AFG. Furthermore the interviews
and focus groups carried out within the stakeholders’ own environment, provided firsthand
evidence of the effects of AFG.
Voluntary groups carrying out their own impact assessment may be limited by resources,
which may reduce the number of stakeholders that can be interviewed. However, appropriate
selection will mean a sufficient range can be interviewed. The combination of quantitative and
qualitative data has meant that the work carried out by AFG can be assessed reasonably fully.
7.4.3 Social capital framework
Problems did arise with the use of the social capital framework as some interviewees, aware
that AFG was being studied, tried to relate effects of AFG’s work to positive aspects of their
own work. To some interviewees it did act almost as a tick-list to which they must reply with
something. Despite this it is felt that aspects of the impact would have been missed were it
not for the framework. As can be seen from the findings, there were other types of impact not
covered in the framework that arose from the interviews.
7.4.4 Indicators as measurement tools
This case-study achieved a set of indicators that can then be used to measure the impact of
any organisation. However it could be argued that by quantifying the data one loses the value
that was gained from interviews and focus groups. Nevertheless, in a competitive funding
environment, there may be a need to ‘quantify’ the impact a voluntary or community group
has on the community.
7.4.5 Negative impact
The themes that emerged from the interviews and focus groups were mainly positive.
Negative aspects would have been included if they came up. However it was thought that
since AFG is a relatively new group, negative impacts are less noticeable than positive aspects.
Also, the stakeholders were identified from AFG and may represent a more biased view than
if they had been identified through other means. However, discovering negative impact may
be a useful future development of this approach.