Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

172 8. Principal Components in Regression AnalysisandE[p∑k=m+1l −1ka ka ′ kX ′ y]==p∑k=m+1p∑k=m+1l −1ka ka ′ kX ′ Xβa k a ′ kβ.This last term is, in general, non-zero so that E(˜β) ≠ β. However, ifmulticollinearity is a serious problem, the reduction in variance can besubstantial, whereas the bias introduced may be comparatively small. Infact, if the elements of γ corresponding to deleted components are actuallyzero, then no bias will be introduced.As well as, or instead of, deleting terms from (8.1.8) corresponding tosmall eigenvalues, it is also possible to delete terms for which the correspondingelements of γ are not significantly different from zero. Thequestion of which elements are significantly non-zero is essentially a variableselection problem, with PCs rather than the original predictor variables asvariables. Any of the well-known methods of variable selection for regression(see, for example, Draper and Smith, 1998, Chapter 15) can be used.However, the problem is complicated by the desirability of also deletinghigh-variance terms from (8.1.8).The definition of PC regression given above in terms of equations (8.1.3)and (8.1.4) is equivalent to using the linear model (8.1.1) and estimatingβ by˜β = ∑ Ml −1k a ka ′ kX ′ y, (8.1.12)where M is some subset of the integers 1, 2,...,p. A number of authors consideronly the special case (8.1.10) of (8.1.12), in which M = {1, 2,...,m},but this is often too restrictive, as will be seen in Section 8.2. In the generaldefinition of PC regression, M can be any subset of the first p integers, sothat any subset of the coefficients of γ, corresponding to the complement ofM, can be set to zero. The next section will consider various strategies forchoosing M, but we first note that once again the singular value decomposition(SVD) of X defined in Section 3.5 can be a useful concept (see alsoSections 5.3, 6.1.5, 13.4, 13.5, 13.6, 14.2 and Appendix A1). In the presentcontext it can be used to provide an alternative formulation of equation(8.1.12) and to help in the interpretation of the results of a PC regression.Assuming that n ≥ p and that X has rank p, recall that the SVD writes Xin the formwhereX = ULA ′ ,(i) A and L are as defined earlier in this section;