Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

410 Appendix A. Computation of Principal Componentsderived. It works well if λ 1 ≫ λ 2 , but converges only slowly if λ 1 is not wellseparated from λ 2 . Speed of convergence also depends on the choice of theinitial vector u 0 ; convergence is most rapid if u 0 is close to α 1 ,If λ 1 = λ 2 >λ 3 , a similar argument to that given above shows that asuitably normalized version of u r → α 1 +(κ 2 /κ 1 )α 2 as r →∞.Thus,the method does not lead to α 1 , but it still provides information aboutthe space spanned by α 1 , α 2 . Exact equality of eigenvalues is extremelyunlikely for sample covariance or correlation matrices, so we need not worrytoo much about this case.Rather than looking at all u r ,r=1, 2, 3,...,attention can be restrictedto u 1 , u 2 , u 4 , u 8 ,...(that is Tu 0 , T 2 u 0 , T 4 u 0 , T 8 u 0 ,...) by simply squaringeach successive power of T. This accelerated version of the power methodwas suggested by Hotelling (1936). The power method can be adapted tofind the second, third, ...PCs, or the last few PCs (see Morrison, 1976,p. 281), but it is likely to encounter convergence problems if eigenvaluesare close together, and accuracy diminishes if several PCs are found by themethod. Simple worked examples for the first and later components can befound in Hotelling (1936) and Morrison (1976, Section 8.4) .There are various adaptations to the power method that partially overcomesome of the problems just mentioned. A large number of suchadaptations are discussed by Wilkinson (1965, Chapter 9), although someare not directly relevant to positive-semidefinite matrices such as covarianceor correlation matrices. Two ideas that are of use for such matriceswill be mentioned here. First, the origin can be shifted, that is the matrixT is replaced by T − ρI p , where I p is the identity matrix, and ρ is chosento make the ratio of the first two eigenvalues of T − ρI p much larger thanthe corresponding ratio for T, hence speeding up convergence.A second modification is to use inverse iteration (with shifts), in whichcase the iterations of the power method are used but with (T − ρI p ) −1replacing T. This modification has the advantage over the basic powermethod with shifts that, by using appropriate choices of ρ (different fordifferent eigenvectors), convergence to any of the eigenvectors of T canbe achieved. (For the basic method it is only possible to converge in thefirst instance to α 1 or to α p .) Furthermore, it is not necessary to explicitlycalculate the inverse of T−ρI p , because the equation u r =(T−ρI p ) −1 u r−1can be replaced by (T − ρI p )u r = u r−1 . The latter equation can thenbe solved using an efficient method for the solution of systems of linearequations (see Wilkinson, 1965, Chapter 4). Overall, computational savingswith inverse iteration can be large compared to the basic power method(with or without shifts), especially for matrices with special structure, suchas tridiagonal matrices. It turns out that an efficient way of computingPCs is to first transform the covariance or correlation matrix to tridiagonalform using, for example, either the Givens or Householder transformations(Wilkinson, 1965, pp. 282, 290), and then to implement inverse iterationwith shifts on this tridiagonal form.