Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

170 8. Principal Components in Regression Analysiswhere L is the diagonal matrix whose kth diagonal element is l 1/2k,andl k is defined here, as in Section 3.5, as the kth largest eigenvalue of X ′ X,rather than S. Furthermore, if the regression equation is calculated for PCsinstead of the predictor variables, then the contributions of each transformedvariable (PC) to the equation can be more easily interpreted thanthe contributions of the original variables. Because of uncorrelatedness, thecontribution and estimated coefficient of a PC are unaffected by which otherPCs are also included in the regression, whereas for the original variablesboth contributions and coefficients can change dramatically when anothervariable is added to, or deleted from, the equation. This is especially truewhen multicollinearity is present, but even when multicollinearity is nota problem, regression on the PCs, rather than the original predictor variables,may have advantages for computation and interpretation. However, itshould be noted that although interpretation of the separate contributionsof each transformed variable is improved by taking PCs, the interpretationof the regression equation itself may be hindered if the PCs have no clearmeaning.The main advantage of PC regression occurs when multicollinearities arepresent. In this case, by deleting a subset of the PCs, especially those withsmall variances, much more stable estimates of β can be obtained. To seethis, substitute (8.1.6) into (8.1.5) to giveˆβ = A(Z ′ Z) −1 Z ′ y (8.1.7)= AL −2 Z ′ y= AL −2 A ′ X ′ yp∑= l −1k a ka ′ kX ′ y, (8.1.8)k=1where l k is the kth diagonal element of L 2 and a k is the kth column of A.Equation (8.1.8) can also be derived more directly from ˆβ =(X ′ X) −1 X ′ y,by using the spectral decomposition (see Property A3 of Sections 2.1 and3.1) of the matrix (X ′ X) −1 , which has eigenvectors a k and eigenvalues,k=1, 2,...,p.Making the usual assumption that the elements of y are uncorrelated,each with the same variance σ 2 (that is the variance-covariance matrix ofy is σ 2 I n ), it is seen from (8.1.7) that the variance-covariance matrix of ˆβisl −1kσ 2 A(Z ′ Z) −1 Z ′ Z(Z ′ Z) −1 A ′ = σ 2 A(Z ′ Z) −1 A ′= σ 2 AL −2 A ′= σ 2 p∑k=1l −1k a ka ′ k. (8.1.9)