Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

168 8. Principal Components in Regression Analysistor variables. As the PCs are uncorrelated, there are no multicollinearitiesbetween them, and the regression calculations are also simplified. If all thePCs are included in the regression, then the resulting model is equivalentto that obtained by least squares, so the large variances caused by multicollinearitieshave not gone away. However, calculation of the least squaresestimates via PC regression may be numerically more stable than directcalculation (Flury and Riedwyl, 1988, p. 212).If some of the PCs are deleted from the regression equation, estimatorsare obtained for the coefficients in the original regression equation.These estimators are usually biased, but can simultaneously greatly reduceany large variances for regression coefficient estimators caused by multicollinearities.Principal component regression is introduced in Section 8.1,and strategies for deciding which PCs to delete from the regression equationare discussed in Section 8.2; some connections between PC regressionand other forms of biased regression are described in Section 8.3.Variations on the basic idea of PC regression have also been proposed.One such variation, noted in Section 8.3, allows the possibility that a PCmay be only ‘partly deleted’ from the regression equation. A rather differentapproach, known as latent root regression, finds the PCs of the predictorvariables together with the dependent variable. These PCs can then beused to construct biased regression estimators, which differ from those derivedfrom PC regression. Latent root regression in various forms, togetherwith its properties, is discussed in Section 8.4. A widely used alternativeto PC regression is partial least squares (PLS). This, too, is included inSection 8.4, as are a number of other regression-related techniques thathave connections with PCA. One omission is the use of PCA to detect outliers.Because the detection of outliers is important in other areas as wellas regression, discussion of this topic is postponed until Section 10.1.A topic which is related to, but different from, regression analysis isthat of functional and structural relationships. The idea is, like regressionanalysis, to explore relationships between variables but, unlike regression,the predictor variables as well as the dependent variable may be subjectto error. Principal component analysis can again be used in investigatingfunctional and structural relationships, and this topic is discussed inSection 8.6.Finally in this chapter, in Section 8.7 two detailed examples are givenof the use of PCs in regression, illustrating many of the ideas discussed inearlier sections.8.1 Principal Component RegressionConsider the standard regression model, as defined in equation (3.1.5), thatis,y = Xβ + ɛ, (8.1.1)