Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

8.1. Principal Component Regression 169where y is a vector of n observations on the dependent variable, measuredabout their mean, X is an (n×p) matrix whose (i, j)th element is the valueof the jth predictor (or regressor) variable for the ith observation, againmeasured about its mean, β is a vector of p regression coefficients and ɛ isa vector of error terms; the elements of ɛ are independent, each with thesame variance σ 2 . It is convenient to present the model (8.1.1) in ‘centred’form, with all variables measured about their means. Furthermore, it isconventional in much of the literature on PC regression to assume that thepredictor variables have been standardized so that X ′ X is proportional tothe correlation matrix for the predictor variables, and this convention isfollowed in the present chapter. Similar derivations to those below are possibleif the predictor variables are in uncentred or non-standardized form,or if an alternative standardization has been used, but to save space andrepetition, these derivations are not given. Nor do we discuss the controversythat surrounds the choice of whether or not to centre the variablesin a regression analysis. The interested reader is referred to Belsley (1984)and the discussion which follows that paper.The values of the PCs for each observation are given byZ = XA, (8.1.2)where the (i, k)th element of Z is the value (score) of the kth PC for theith observation, and A is a (p × p) matrix whose kth column is the ktheigenvector of X ′ X.Because A is orthogonal, Xβ can be rewritten as XAA ′ β = Zγ, whereγ = A ′ β. Equation (8.1.1) can therefore be written asy = Zγ + ɛ, (8.1.3)which has simply replaced the predictor variables by their PCs in the regressionmodel. Principal component regression can be defined as the useof the model (8.1.3) or of the reduced modely = Z m γ m + ɛ m , (8.1.4)where γ m is a vector of m elements that are a subset of elements of γ,Z m is an (n × m) matrix whose columns are the corresponding subset ofcolumns of Z, andɛ m is the appropriate error term. Using least squares toestimate γ in (8.1.3) and then finding an estimate for β from the equationˆβ = Aˆγ (8.1.5)is equivalent to finding ˆβ by applying least squares directly to (8.1.1).The idea of using PCs rather than the original predictor variables is notnew (Hotelling, 1957; Kendall, 1957), and it has a number of advantages.First, calculating ˆγ from (8.1.3) is more straightforward than finding ˆβfrom (8.1.1) as the columns of Z are orthogonal. The vector ˆγ isˆγ =(Z ′ Z) −1 Z ′ y = L −2 Z ′ y, (8.1.6)