Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

6 1. Introductioncould be used to specify zero correlation between α ′ 1x and α ′ 2x. Choosingthe last of these (an arbitrary choice), and noting that a normalizationconstraint is again necessary, the quantity to be maximized isα ′ 2Σα 2 − λ(α ′ 2α 2 − 1) − φα ′ 2α 1 ,where λ, φ are Lagrange multipliers. Differentiation with respect to α 2givesΣα 2 − λα 2 − φα 1 = 0and multiplication of this equation on the left by α ′ 1 givesα ′ 1Σα 2 − λα ′ 1α 2 − φα ′ 1α 1 =0,which, since the first two terms are zero and α ′ 1α 1 = 1, reduces to φ =0.Therefore, Σα 2 − λα 2 = 0, or equivalently (Σ − λI p )α 2 = 0, soλ is oncemore an eigenvalue of Σ, andα 2 the corresponding eigenvector.Again, λ = α ′ 2Σα 2 ,soλ is to be as large as possible. Assuming thatΣ does not have repeated eigenvalues, a complication that is discussed inSection 2.4, λ cannot equal λ 1 . If it did, it follows that α 2 = α 1 , violatingthe constraint α ′ 1α 2 = 0. Hence λ is the second largest eigenvalue of Σ,and α 2 is the corresponding eigenvector.As stated above, it can be shown that for the third, fourth, ..., pthPCs, the vectors of coefficients α 3 , α 4 ,...,α p are the eigenvectors of Σcorresponding to λ 3 ,λ 4 ,...,λ p , the third and fourth largest, ..., and thesmallest eigenvalue, respectively. Furthermore,var[α ′ kx] =λ kfor k =1, 2,...,p.This derivation of the PC coefficients and variances as eigenvectors andeigenvalues of a covariance matrix is standard, but Flury (1988, Section 2.2)and Diamantaras and Kung (1996, Chapter 3) give alternative derivationsthat do not involve differentiation.It should be noted that sometimes the vectors α k are referred toas ‘principal components.’ This usage, though sometimes defended (seeDawkins (1990), Kuhfeld (1990) for some discussion), is confusing. It ispreferable to reserve the term ‘principal components’ for the derived variablesα ′ k x, and refer to α k as the vector of coefficients or loadings for thekth PC. Some authors distinguish between the terms ‘loadings’ and ‘coefficients,’depending on the normalization constraint used, but they will beused interchangeably in this book.1.2 A Brief History of Principal ComponentAnalysisThe origins of statistical techniques are often difficult to trace. Preisendorferand Mobley (1988) note that Beltrami (1873) and Jordan (1874)