Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

A.1. Numerical Calculation of Principal Components 409Finding PCs reduces to finding the eigenvalues and eigenvectors of apositive-semidefinite matrix. We now look briefly at some of the possiblealgorithms that can be used to solve such an eigenproblem.The Power MethodA form of the power method was described by Hotelling (1933) in hisoriginal paper on PCA, and an accelerated version of the technique waspresented in Hotelling (1936). In its simplest form, the power method is atechnique for finding the largest eigenvalue and the corresponding eigenvectorof a (p × p) matrix T. The idea is to choose an initial p-elementvector u 0 , and then form the sequenceu 1 = Tu 0u 2 = Tu 1 = T 2 u 0..u r = Tu r−1 = T r u 0.lf α 1 , α 2 ,...,α p are the eigenvectors of T, then they form a basis forp-dimensional space, and we can write, for arbitrary u 0 ,p∑u 0 = κ k α kk=1for some set of constants κ 1 ,κ 2 ,...,κ p . Thenp∑p∑u 1 = Tu 0 = κ k Tα k = κ k λ k α k ,k=1where λ 1 ,λ 2 ,...,λ p are the eigenvalues of T. Continuing, we get for r =2, 3,...p∑u r = κ k λ r kα kk=1and(u r(κ 1 λ r 1 ) = α 1 + κ ( ) r2 λ2α 2 + ···+ κ ( ) r )p λpα p .κ 1 λ 1 κ 1 λ 1Assuming that the first eigenvalue of T is distinct from the remainingeigenvalues, so that λ 1 >λ 2 ≥ ··· ≥ λ p , it follows that a suitably normalizedversion of u r → α 1 as r →∞. It also follows that the ratios ofcorresponding elements of u r and u r−1 → λ 1 as r →∞.The power method thus gives a simple algorithm for finding the first(largest) eigenvalue of a covariance or correlation matrix and its correspondingeigenvector, from which the first PC and its variance can be.k=1