Is headspace making a difference to young people’s lives?

2. Evaluation Methodology
initial K10 score. For this category, significance is measured at the 5% level. A clinically significant
change occurs when the change in K10 is both reliably significant and also moves the headspace
client below or above the threshold K10 that represents a benchmark for the general population. A
clinically significant improvement can be regarded as a change sufficient to revert the client to a level
of psychological functioning that is consistent with that of a functional population.
Thresholds for clinical significance require information on the distribution of K10 scores for a general
population as a comparison. The CSC analysis presented in this report derives threshold K10 scores
from information in the comparison surveys. The analysis presented in Table 4.1 shows the mean
change in K10 scores for each category of change. This analysis shows for example that those
young people in the clinically significant improvement category had a mean reduction between first
and last recorded K10 of 14.6 points; those in the insignificant improvement category had a mean
reduction of 4.8 K10 points; and those in the clinically significant decline category had a mean
increase in psychological distress as measured by the K10 of 14.1 points. Results of these methods
of analyses (DID and CSC) are reported in Chapter 4.
2.5 Evaluation Limitations
Like all evaluations of complex human service programs, a number of unanticipated challenges were
encountered during the course of the evaluation. It is important to consider these challenges and
limitations when interpreting the findings presented throughout this report. This section provides a
summary of the issues that were of particular significance for the evaluation.
Attribution
Attribution is a challenge in any evaluation, particularly those without an experimental design such as
a fully specified Randomised Control Trial (RCT). Due to the diversity of the headspace treatment,
clients, and service providers, it is neither feasible nor reasonable for an RCT to be conducted to
assess the overall impact of headspace services on young people accessing headspace centres
across Australia. Given this, one approach to attribution is to exploit the existence of a ‘natural
experiment’. This method seeks to compare the relative progress of a ‘comparison’ group of young
people who can be considered similar in their economic and social circumstances, and with similar
presenting conditions, to the headspace treatment group. To achieve a closer degree of alignment
between treatment and comparison groups, the evaluation team matched the two samples on a set
of observed characteristics using propensity score methods (see Appendix C for further information).
This can provide some degree of identification of effectiveness in principal, although there are
limitations with this approach relative to full RCT methods.
Comparison groups
As outlined above, the young people surveys were completed by a sample of headspace clients
and two comparison groups: a sample of 12-17 year olds who participated in Young Minds Matter, a
national survey of children’s health and wellbeing; and a sample of 18-25 year olds sourced through
a national online panel. For purposes of the evaluation, the comparison group was separated into a
‘no treatment’ group of young people from the general population who had not accessed headspace
or any other treatment for a mental health or drug and alcohol condition, and an ‘other treatment’
group who received alternative forms of mental health care between the two waves of data collection.
Due to data limitations, the evaluators are not able to assess the type, intensity or duration of the
alternative treatment received by young people in the ‘other treatment’ group.
In order to attribute changes in the intervention group to headspace, the comparison group should be
as representative as possible of the headspace population in terms of demography and wave 1 levels
of psychological distress. Comparative analysis of demographic data showed that the 18-25 year
old comparison group was somewhat different to the headspace population. To address this issue,
the evaluators undertook propensity score matching of survey groups. This method allows for a
closer comparison between the ‘headspace treatment’ and comparison cohorts but it is not a perfect
comparison. The evaluators were unable to match on more than four variables without significant
differences in distributions, and the method does not account for unobserved differences between
the treatment and comparison cohorts.
Social Policy Research Centre 2015
headspace Evaluation Final Report
16