Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

show that X = ULA ′ .⎡ULA ′ = U⎢⎣==r∑k=13.5. The Singular Value Decomposition 45l 1/21 a ′ 1l 1/22 a ′ 2..lr1/2 a ′ r⎤⎥⎦l −1/2kXa k l 1/2ka ′ k =p∑Xa k a ′ k.k=1r∑Xa k a ′ kThis last step follows because a k ,k=(r+1), (r+2),...,p, are eigenvectorsof X ′ X corresponding to zero eigenvalues. The vector Xa k is a vector ofscores on the kth PC; the column-centering of X and the zero variance ofthe last (p−r) PCs together imply that Xa k = 0, k =(r+1), (r+2),...,p.Thusp∑ULA ′ = X a k a ′ k = X,k=1as required, because the (p × p) matrix whose kth column is a k , isorthogonal, and so has orthonormal rows.The importance of the SVD for PCA is twofold. First, it provides acomputationally efficient method of actually finding PCs (see AppendixA1). It is clear that if we can find U, L, A satisfying (3.5.1), then A andL will give us the eigenvectors and the square roots of the eigenvalues ofX ′ X, and hence the coefficients and standard deviations of the principalcomponents for the sample covariance matrix S. As a bonus we also get inU scaled versions of PC scores. To see this multiply (3.5.1) on the rightby A to give XA = ULA ′ A = UL, asA ′ A = I r . But XA is an (n × r)matrix whose kth column consists of the PC scores for the kth PC (see(3.1.2) for the case where r = p). The PC scores z ik are therefore given byz ik = u ik l 1/2k, i =1, 2,...,n, k =1, 2,...,r,or, in matrix form, Z = UL, orU = ZL −1 . The variance of the scoreslfor the kth PC is k(n−1) ,k=1, 2,...,p. [Recall that l k here denotes thek=1l k(n−1)kth eigenvalue of X ′ X,sothatthekth eigenvalue of S is .] Thereforethe scores given by U are simply those given by Z, but scaled to have1variance(n−1). Note also that the columns of U are the eigenvectors ofXX ′ corresponding to non-zero eigenvalues, and these eigenvectors are ofpotential interest if the rôles of ‘variables’ and ‘observations’ are reversed.A second virtue of the SVD is that it provides additional insight intowhat a PCA actually does, and it gives useful means, both graphical and