Evaluation of the Australian Wage Subsidy Special Youth ...

136
4.8.2 Mean standardised bias statistic
The distributions for the remaining comparisons when compared to the SYETP group are
not perfectly balanced in terms of the separate independent variables after propensity
score matching. This is because the propensity score allows some variables to be less
well matched for a case, while others variables are well matched, leading to an overall
weighting scale across each of these. A measure that summarises the overall balance is
the standardized mean bias. For each variable included in the probit to estimate the
propensity score, the measure is calculated as:
100*[ | µ treated - µ comparison | / √{( var treated + var comparison )/2} ]
where µ is the mean and var is the variance observed for the covariates in the sample.
The summary measure of the distribution of bias is found as the mean standardised bias
across all the variables. 89 This measure can be used to gauge the matching performance
(Lechner (2000)). It can be interpreted as bias as a percentage of the standard deviation.
A smaller average bias is generally considered better. It is clear that a bias of zero is ideal,
but the tolerance level for any non-zero standardized bias is not defined. Matching
generally strongly improves mean bias. A poor match quality would indicate that the
matched groups are not genuinely balanced on the observed characteristics and that no
outcome comparisons based on these matches would be justified (Frölich et al. (2000):
39). However there is no clear gauge as to what level of standardized bias indicates a
match is poor.
Comparison with other studies, which report the standardized bias, gives another gauge
as to whether the levels are acceptable. Some recent studies that report balance measures
are Sianesi (2001), Larsson (2000), Frölich et al. (2000), and Gerfin and Lechner (2000).
Sianesi (2001) p47 appendix D found in Swedish administrative data analyses for 31,975
cases a mean standardized bias of 0.86, but some variables were poor such as age at 2.86.
This was a large data set, which enhances the ability to match. Larsson (2000) p27 and
89 The use of the term bias here is the statistical sense, and refers to the differences in the conditioning
variables used for matching and not bias in an estimator for a parameter.