Is headspace making a difference to young people’s lives?

2. Evaluation Methodology
2.4 Statistical analysis techniques
The outcomes analysis draws upon data contained within the hCSA and the young people surveys.
The analysis has been conducted using two distinct approaches. These are:
• Difference-in-difference (DID) approach, and
• Clinically significant change (CSC) method.
Further information on how these methods have been specifically applied is contained in Appendix C;
however, summarised information is also provided here.
2.4.1 The Difference-in-difference Method
The young people survey data has been analysed using a difference-in-difference (DID) approach.
This non-experimental method is commonly used to evaluate the impact of programs or interventions.
The simplest design for a DID analysis calculates the effect of a treatment (headspace) on an
outcome (for example psychological distress). This is done through a comparison of the average
change over time in the outcome variable for the treatment group to the average change over time for
the comparison group.
The specific objective of the DID method for the evaluation is to assess the changes in young
people’s mental health, physical health, drug and alcohol use and social inclusion outcomes after
using headspace services relative to other comparable young people that did not receive treatment at
a headspace centre. A scoping analysis of the survey data demonstrated differences in the profiles
of the ‘headspace treatment’ group compared to those captured within the comparison surveys,
illustrating that the headspace survey clients were quite different to the general population. To
address this limitation, the evaluators sought to match the groups through propensity score matching
– a statistical matching technique that aims to better align intervention and comparison cohorts.
Two groups were extracted from the comparison surveys to match the ‘headspace treatment’ group,
those that received some other mental health treatment (the ‘other treatment’ group) and those
that received no treatment (the ‘no treatment’ group). A number of variables were tested to align
the groups, with four key variables - K10 score, age, gender, and days out of role - confirmed as
benchmarks for the matching technique. The propensity score matching has resulted in a smaller
sample but closer alignment between the groups of interest. Further information on the treatment
groups and propensity score matching, including the age and sex distributions of matched groups is
available in Appendix C.
The results of the DID analyses are presented in Chapter 4 below. Difference-in-difference estimates
are defined as the difference in the average outcome in the ‘headspace treatment’ group at two
points in time, that is at wave 1 and wave 2 data collection, minus the difference in the average
outcome in the matched comparison groups (‘other treatment’ and ‘no treatment’ groups). Six key
outcomes variables contained within the survey data are used to assess changes in mental and
physical health, social and vocational participation, and alcohol and drug use (see description in
Table 2.3 below).
Chapter 4 also reports on whether the differences in outcomes between the matched groups
are statistically significant (determined by using an orthodox t-test). Finally, effect sizes are also
reported. Effect sizes can be expressed in a number of ways, with Cohen’s d commonly reported as
a standard indicator in clinical evaluation. The Cohen effect size measure presents a standardised
difference in means across the course of an intervention (that is, the ratio of mean difference to a
pooled standard deviation measure).
Social Policy Research Centre 2015
headspace Evaluation Final Report
14