Evaluation of the Australian Wage Subsidy Special Youth ...

21
this case, if there was no data loss such as due to loss of contact or other confounding
problems, then the experiment might avoid the selection problem. Random assignment
creates a control group that comprises individuals, which within sampling variation
should have identical distributions of observable and unobservable characteristics to
those in the treatment group. Random participation can thus defeat the potential for
selection bias. In the absence of this, non-experimental methods for evaluation should be
used. However, data problems of non-response, or treatment differing from assigned
treatment, can re-introduce selection problems into even experimental data. For example
Heckman et al. (1999) pp1905 -1914 discuss the data problems of economic program
experimental data, such as sample attrition. Fay (1996) Table 4a p47 also lists some
problems of experiments as randomisation bias, sample contamination, treatment
contamination, site self-selection, substitution bias, crossover bias, and program entry
effects.
Heckman et al. (1999) examine in detail the form of evaluation counterfactual estimated
using various evaluation methods. Both the Heckman selection model and matching
methods produce the parameter corresponding to the mean effect of treatment on the
treated, which Heckman et al. (1999) derive fully. Hujer and Caliendo (2000) p10 note
this parameter answers the question “what is the expected outcome gain to individuals
who received treatment, to the hypothetical situation they had not received it?”. Heckman
et al. (1999) and Heckman et al. (1997, 1998) point out that this question focuses directly
on actual participants, and can help decide whether the program is a success. However,
the identifying assumptions used to estimate the unobserved counterfactual term from the
observed non-participant outcomes, that are required for the Heckman selection model,
differ from those required for matching to produce a valid estimation of the mean effect
of treatment on the treated. Each of these modelling methods and their assumptions is
treated in detail in the later chapters of this research.