Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

3.2. Geometric Properties of Sample Principal Components 33the last q columns of a matrix whose kth column is the kth eigenvectorof (X ′ X) −1 . Furthermore, (X ′ X) −1 has the same eigenvectors as X ′ X, exceptthat their order is reversed, so that B q must have columns equal tothe first q eigenvectors of X ′ X. As this holds for q =1, 2,...,p, PropertyA7 is proved.✷This property seems to imply that replacing the predictor variables in aregression analysis by their first few PCs is an attractive idea, as those PCsomitted have coefficients that are estimated with little precision. The flaw inthis argument is that nothing in Property A7 takes account of the strengthof the relationship between the dependent variable y and the elements ofx, orbetweeny and the PCs. A large variance for ˆγ k ,thekth element ofγ, and hence an imprecise estimate of the degree of relationship between yand the kth PC, z k , does not preclude a strong relationship between y andz k (see Section 8.2). Further discussion of Property A7 is given by Fombyet al. (1978).There are a number of other properties of PCs specific to the samplesituation; most have geometric interpretations and are therefore dealt within the next section.3.2 Geometric Properties of Sample PrincipalComponentsAs with the algebraic properties, the geometric properties of Chapter 2are also relevant for sample PCs, although with slight modifications to thestatistical implications. In addition to these properties, the present sectionincludes a proof of a sample version of Property A5, viewed geometrically,and introduces two extra properties which are relevant to sample, but notpopulation, PCs.Property G1 is still valid for samples if Σ is replaced by S. The ellipsoidsx ′ S −1 x = const no longer have the interpretation of being contours ofconstant probability, though they will provide estimates of such contoursif x 1 , x 2 ,...,x n are drawn from a multivariate normal distribution. Reintroducinga non-zero mean, the ellipsoids(x − ¯x) ′ S −1 (x − ¯x) = constgive contours of equal Mahalanobis distance from the sample mean ¯x.Flury and Riedwyl (1988, Section 10.6) interpret PCA as successively findingorthogonal directions for which the Mahalanobis distance from thedata set to a hypersphere enclosing all the data is minimized (see Sections5.3, 9.1 and 10.1 for discussion of Mahalanobis distance in a varietyof forms).