Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

2.1. Optimal Algebraic Properties of Population Principal Components 17It follows that although Property A5 is stated as an algebraic property,it can equally well be viewed geometrically. In fact, it is essentially thepopulation equivalent of sample Property G3, which is stated and provedin Section 3.2. No proof of the population result A5 will be given here; Rao(1973, p. 591) outlines a proof in which y is replaced by an equivalent setof uncorrelated linear functions of x, and it is interesting to note that thePCs are the only set of p linear functions of x that are uncorrelated andhave orthogonal vectors of coefficients. This last result is prominent in thediscussion of Chapter 11.A special case of Property A5 was pointed out in Hotelling’s (1933)original paper. He notes that the first PC derived from a correlation matrixis the linear function of x that has greater mean square correlation withthe elements of x than does any other linear function. We return to thisinterpretation of the property, and extend it, in Section 2.3.A modification of Property A5 can be introduced by noting that if x ispredicted by a linear function of y = B ′ x, then it follows from standardresults from multivariate regression (see, for example, Mardia et al., 1979,p. 160), that the residual covariance matrix for the best such predictor isΣ x − Σ xy Σ −1y Σ yx , (2.1.11)where Σ x = Σ, Σ y = B ′ ΣB, as defined before, Σ xy is the matrix whose(j, k)th element is the covariance between the jth element of x and thekth element of y, andΣ yx is the transpose of Σ xy .NowΣ yx = B ′ Σ,andΣ xy = ΣB, so (2.1.11) becomesΣ − ΣB(B ′ ΣB) −1 B ′ Σ. (2.1.12)The diagonal elements of (2.1.12) are σj 2 ,j=1, 2,...,p, so, from PropertyA5, B = A q minimizesp∑σj 2 =tr[Σ − ΣB(B ′ ΣB) −1 B ′ Σ].j=1A derivation of this result in the sample case, and further discussion of it,is provided by Jong and Kotz (1999).An alternative criterion is ‖Σ − ΣB(B ′ ΣB) −1 B ′ Σ‖, where ‖·‖ denotesthe Euclidean norm of a matrix and equals the square root of the sum ofsquares of all the elements in the matrix. It can be shown (Rao, 1964) thatthis alternative criterion is also minimized when B = A q .This section has dealt with PCs derived from covariance matrices. Manyof their properties are also relevant, in modified form, for PCs based oncorrelation matrices, as discussed in Section 2.3. That section also containsa further algebraic property which is specific to correlation matrix-basedPCA.