Evaluation of the Australian Wage Subsidy Special Youth ...

210
reasonable for smaller calipers, for the wider calipers of 0.01, 0.02 and 0.05, the
statistical significance falls off dramatically. This would indicate that applying a wider
caliper was far more consequential in the weighted data. Compared to the former
unweighted results, there is a much greater variation in the size of employment effect
estimated. However, none of the new PSM results for the employment effect have
statistical significance at normal test levels. To an extent this would be due to the use of
weights as the variance estimate is then less efficient. When the weights are used, the
more conservative estimate of the variance is used in order to account for the fact that it
is assumed the missing data follows the same pattern as that of the non-missing data.
The effectiveness of the matching as assessed by the standardized bias measure gives a
slightly poorer performance than for the former unweighted results. The weighted
matches have standardized mean biases which are between 11 and 12 in magnitude,
whereas before they were between 9 and 10. This change in size is very slight. As
discussed in the preceding chapter, this would indicate that caution should be applied in
accepting the performance of the match. The mean difference in the propensity scores
after matching is very small, which would indicate close matching as a small mean
difference is consistent with good matches. The number of comparison group
observations used for the matching has slightly risen in the weighted results, compared to
the unweighted, which would indicate slightly less reliance on a few comparison cases.
However, taken as a whole, the effectiveness of the weighted match is not too different
from that of the unweighted matching.
A graphical summary of the matched propensity scores for the 0.001 caliper is shown in
Figure 6.8. Although a kernel density plot can only give a rough guide, the distributions
of the matched propensity scores overlap very well. It would seem that at the upper and
lower extremes, the overlap is better, but in the middle regions of the distributions there
are two gaps. In the first gap, at lower propensities, the untreated lie outside the SYETP,
while in the second gap, the SYETP lie outside the untreated. In these gaps, the
difference in the propensity scores of the treated and comparison would be greater, and in
these gap regions the match is slightly poorer.