Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

182 8. Principal Components in Regression Analysisby rescaling and then using varimax rotation (see Chapter 7), which hasthe effect of transforming the PCs to a different set of uncorrelated variables.These variables are, like the PCs, linear functions of the original(p + 1) variables, but their coefficients are mostly close to zero or a longway from zero, with relatively few intermediate values. There is no guarantee,in general, that any of the new variables will have particularly large orparticularly small variances, as they are chosen by simplicity of structureof their coefficients, rather than for their variance properties. However, ifonly one or two of the coefficients for y are large, as should often happenwith varimax rotation, then Hawkins (1973) shows that the correspondingtransformed variables will have very small variances, and therefore suggestlow-variance relationships between y and the predictor variables. Otherpossible regression equations may be found by substitution of one subset ofpredictor variables in terms of another, using any low-variability relationshipsbetween predictor variables that are suggested by the other rotatedPCs.The above technique is advocated by Hawkins (1973) and by Jeffers(1981) as a means of selecting which variables should appear in the regressionequation (see Section 8.5), rather than as a way of directly estimatingtheir coefficients in the regression equation, although the technique couldbe used for the latter purpose. Daling and Tamura (1970) also discussedrotation of PCs in the context of variable selection, but their PCs were forthe predictor variables only.In a later paper, Hawkins and Eplett (1982) propose another variant of latentroot regression one which can be used to efficiently find low-variabilityrelationships between y and the predictor variables, and which also can beused in variable selection. This method replaces the rescaling and varimaxrotation of Hawkins’ earlier method by a sequence of rotations leading toa set of relationships between y and the predictor variables that are simplerto interpret than in the previous method. This simplicity is achievedbecause the matrix of coefficients defining the relationships has non-zeroentries only in its lower-triangular region. Despite the apparent complexityof the new method, it is also computationally simple to implement. Thecovariance (or correlation) matrix ˜Σ of y and all the predictor variables isfactorized using a Cholesky factorization˜Σ = DD ′ ,where D is lower-triangular. Then the matrix of coefficients defining therelationships is proportional to D −1 , which is also lower-triangular. To findD it is not necessary to calculate PCs based on ˜Σ, which makes the linksbetween the method and PCA rather more tenuous than those betweenPCA and latent root regression. The next section discusses variable selectionin regression using PCs, and because all three variants of latent rootregression described above can be used in variable selection, they will allbe discussed further in that section.