Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

186 8. Principal Components in Regression Analysisproduces a large number of near-zero coefficients for the rotated variables,so that in low-variance relationships involving y (if such low-variance relationshipsexist) only a subset of the predictor variables will have coefficientssubstantially different from zero. This subset forms a plausible selection ofvariables to be included in a regression model. There may be other lowvariancerelationships between the predictor variables alone, again withrelatively few coefficients far from zero. If such relationships exist, and involvesome of the same variables as are in the relationship involving y,then substitution will lead to alternative subsets of predictor variables.Jeffers (1981) argues that in this way it is possible to identify all good subregressionsusing Hawkins’ (1973) original procedure. Hawkins and Eplett(1982) demonstrate that their newer technique, incorporating Cholesky factorization,can do even better than the earlier method. In particular, for anexample that is analysed by both methods, two subsets of variables selectedby the first method are shown to be inappropriate by the second.Principal component regression and latent root regression may also beused in an iterative manner to select variables. Consider, first, PC regressionand suppose that ˜β given by (8.1.12) is the proposed estimator for β.Then it is possible to test whether or not subsets of the elements of ˜β aresignificantly different from zero, and those variables whose coefficients arefound to be not significantly non-zero can then be deleted from the model.Mansfield et al. (1977), after a moderate amount of algebra, construct theappropriate tests for estimators of the form (8.1.10), that is, where thePCs deleted from the regression are restricted to be those with the smallestvariances. Provided that the true coefficients of the deleted PCs are zeroand that normality assumptions are valid, the appropriate test statisticsare F -statistics, reducing to t-statistics if only one variable is considered ata time. A corresponding result will also hold for the more general form ofestimator (8.1.12).Although the variable selection procedure could stop at this stage, it maybe more fruitful to use an iterative procedure, similar to that suggested byJolliffe (1972) for variable selection in another (non-regression) context (seeSection 6.3, method (i)). The next step in such a procedure is to performa PC regression on the reduced set of variables, and then see if any furthervariables can be deleted from the reduced set, using the same reasoningas before. This process is repeated, until eventually no more variables aredeleted. Two variations on this iterative procedure are described by Mansfieldet al. (1977). The first is a stepwise procedure that first looks for thebest single variable to delete, then the best pair of variables, one of which isthe best single variable, then the best triple of variables, which includes thebest pair, and so on. The procedure stops when the test for zero regressioncoefficients on the subset of excluded variables first gives a significant result.The second variation is to delete only one variable at each stage, and thenrecompute the PCs using the reduced set of variables, rather than allowingthe deletion of several variables before the PCs are recomputed. According