Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

46 3. Properties of Sample Principal Componentsalgebraic, of representing the results of a PCA. This has been recognizedin different contexts by Mandel (1972), Gabriel (1978), Rasmusson et al.(1981) and Eastment and Krzanowski (1982), and will be discussed furtherin connection with relevant applications in Sections 5.3, 6.1.5, 9.3,13.4, 13.5 and 13.6. Furthermore, the SVD is useful in terms of bothcomputation and interpretation in PC regression (see Section 8.1 and Mandel(1982)) and in examining the links between PCA and correspondenceanalysis (Sections 13.1 and 14.2).In the meantime, note that (3.5.1) can be written element by element asx ij =r∑k=1u ik l 1/2ka jk , (3.5.2)where u ik , a jk are the (i, k)th, (j, k)th elements of U, A, respectively, andis the kth diagonal element of L. Thusx ij can be split into partsl 1/2ku ik l 1/2ka jk , k =1, 2,...,r,corresponding to each of the first r PCs. If only the first m PCs are retained,thenm˜x ij =m∑k=1u ik l 1/2ka jk (3.5.3)provides an approximation to x ij . In fact, it can be shown (Gabriel, 1978;Householder and Young, 1938) that m˜x ij gives the best possible rank mapproximation to x ij , in the sense of minimizingn∑i=1 j=1p∑( m x ij − x ij ) 2 , (3.5.4)where m x ij is any rank m approximation to x ij . Another way of expressingthis result is that the (n × p) matrix whose (i, j)th element is m˜x ijminimizes ‖ m X − X‖ over all (n × p) matrices m X with rank m. ThustheSVD provides a sequence of approximations to X of rank 1, 2,...,r,whichminimize the Euclidean norm of the difference between X and the approximationm X. This result provides an interesting parallel to the result givenearlier (see the proof of Property G4 in Section 3.2): that the spectral decompositionof X ′ X provides a similar optimal sequence of approximationsof rank 1, 2,...,r to the matrix X ′ X. Good (1969), in a paper extolling thevirtues of the SVD, remarks that Whittle (1952) presented PCA in termsof minimizing (3.5.4).Finally in this section we note that there is a useful generalization of theSVD, which will be discussed in Chapter 14.