Measuring Impact - Nicva

42 Measuring Impact – Case-studies of impact assessment
These questions were used at an early stage to inform the design and focus of face-to-face interviews
with service providers and service users. Although this framework would have inevitably
influenced the balance of the interview discussions, which were otherwise left unstructured, they
were not allowed to restrict the emerging themes and interviewees’ perceptions of impact.
During the interview stage, the research assistants were matched to service users of similar
ethnicity and gender, while the research co-ordinator conducted interviews across these groups.
At the end of the interview stage, a group feedback session was held with the research assistants.
A consensus was sought regarding the common themes and issues that the team had
identified and agreed. These findings were used to build a picture of the main types of impact
relevant to service users.
6.2.3 Stage three: Assessment of the findings
The purpose of developing thematic areas or indicators is to assess the difference made by a
particular organisation, taking into account the context in which it operates. Moreover, as the
themes are developed largely from the perspectives of the service user, notions of ‘effectiveness’
and ‘quality’ take precedence over ‘efficiency’.
While thematic areas were identified in the previous stage, factors outside the control of the
project restricted the time needed to develop from these themes a set of indicators specific to
the project. In order to do this, the thematic areas would need to be taken back to the original
service users and providers, who could elaborate or contest the analysis of the
researchers. A consensus would then be sought on these main types of impact, which would
be used as a basis for discussion to establish project-specific impact indicators. This would
have involved focus groups with the service providers and service users who had originally
taken part in the face-to-face interviews.
A logical next step would be to apply these indicators to a wider sample of stakeholders as
identified through the mapping exercise in stage one, to allow an assessment of the case-study
organisation’s impact.
Findings 6.3
This section summarises the key findings that emerged by comparing the viewpoints of staff
in the service providers with those of service users. It became obvious that:
• impact is perceived by stakeholders at three distinct levels – personal, community
and organisation level;
• the extent to which the service users concerned themselves with impact on their
community (going beyond impact on their personal situation) reflects gender and
cultural differences;
• each stakeholder appeared to have a clear perception of the organisational context
within which particular organisations operate. This influenced the kinds of changes
that front-line staff attempted to make as well as the expectations of service users.