Evaluation of the Australian Wage Subsidy Special Youth ...

123
employment and programme participation. As such, it is considered that at least with
regard to the breadth of characteristics controlled for, the CIA has been satisfied.
Some measures of fit for the probit model as a whole are considered. However
interpretation of such measures of fit is limited. Scalar measures of fit have a heuristic
nature and there is no convincing evidence that a model which maximizes the value of a
given measure gives a model that is optimal in any sense other than the optimisation of
the particular measure of fit. However some of these measures such as the AIC can be
useful in comparing competing models, which is considered in later chapters. The Akaike
Information Criterion (AIC) is a measure that can be used to compare models across
different samples and non-nested models. All else being equal, the model with the
smaller AIC is judged to be better. Here, the value is 0.57. Tukey’s ‘goodness of fit’ test
is applied to check whether the specification of the probit is suitable and the results
indicate no problems. 81 The chi squared distributed likelihood ratio test of the hypothesis
that all coefficients except the intercept are equal to zero is 107.57 and results in this
hypothesis being rejected. Mcfadden’s pseudo R 2 , sometimes known as the likelihood
ratio index, also compares the model to a model with just an intercept. This tests the same
hypothesis and equals zero if the model is equivalent to the intercept model (but can
never precisely equal one). Here the value is 0.149, and indicates the model is reasonable.
A summary measure of the fit of the predictions shows that 92 per cent were correctly
estimated. 82 Thus the predictive power of the model appears acceptable within the
context of the data. However, prediction rates are not ideal measures for assessing the
propensity score as if the prediction rate were perfect at one, then there would be no
common support. As such, the value of these measures for critically assessing the
propensity are low and judging the included variables, together with the estimated model
parameters, in the context of the theory motivating the analysis is possibly more
important, and examining the estimated propensity scores.
81 A probit of SYETP is run on just the predictions, and the predictions squared, sourced from the model of
SYETP. If the model is specified correctly, the predictions squared should have no explanatory power – i.e.
are statistically insignificant. The base reference is Tukey (1949). The results give a t value close to zero,
which is clearly not significant.
82 A fitted probability exceeding 0.5 is taken to indicate a predicted participation in SYETP, and these
predicted responses are compared to the actual participants/non-participants in SYETP to check which
cases the model correctly predicted.