Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

44 3. Properties of Sample Principal Componentsnear-constant relationship has certainly been identified by the last PC, butnot in an easily interpretable form. However, a simple interpretation can berestored if the standardized variables are replaced by the original variablesby setting x ∗ j = x j/s 1/2jj, where s1/2 jj is the sample standard deviation of x j .When this is done, the last PC becomes (rounding coefficients to the nearestinteger) 11x 1 +11x 2 +11x 3 +11x 4 . The correct near-constant relationshiphas therefore been discovered exactly, to the degree of rounding used, bythe last PC.3.5 The Singular Value DecompositionThis section describes a result from matrix theory, namely the singularvalue decomposition (SVD), which is relevant to PCA in several respects.Given an arbitrary matrix X of dimension (n × p), which for our purposeswill invariably be a matrix of n observations on p variables measured abouttheir means, X can be writtenwhereX = ULA ′ , (3.5.1)(i) U, A are (n × r), (p × r) matrices, respectively, each of which hasorthonormal columns so that U ′ U = I r , A ′ A = I r ;(ii) L is an (r × r) diagonal matrix;(iii) r is the rank of X.To prove this result, consider the spectral decomposition of X ′ X. The last(p − r) terms in (3.1.4) and in the corresponding expression for X ′ X arezero, since the last (p − r) eigenvalues are zero if X, and hence X ′ X,hasrank r. Thus(n − 1) S = X ′ X = l 1 a 1 a ′ 1 + l 2 a 2 a ′ 2 + ···+ l r a r a ′ r.[Note that in this section it is convenient to denote the eigenvalues of X ′ X,rather than those of S, asl k ,k=1, 2,...,p.] Define A to be the (p × r)matrix with kth column a k , define U as the (n × r) matrix whose kthcolumn isu k = l −1/2kXa k , k =1, 2,...,r,and define L to be the (r × r) diagonal matrix with kth diagonal element. Then U, L, A satisfy conditions (i) and (ii) above, and we shall nowl 1/2k