Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)(518s)

14 2. Properties of Population Principal ComponentsThis result will prove to be useful later. Looking at diagonal elements,we see thatp∑var(x j )= λ k αkj.2k=1However, perhaps the main statistical implication of the result is that notonly can we decompose the combined variances of all the elements of xinto decreasing contributions due to each PC, but we can also decomposethe whole covariance matrix into contributions λ k α k α ′ kfrom each PC. Althoughnot strictly decreasing, the elements of λ k α k α ′ kwill tend to becomesmaller as k increases, as λ k decreases for increasing k, whereas the elementsof α k tend to stay ‘about the same size’ because of the normalizationconstraintsα ′ kα k =1,k =1, 2,...,p.Property Al emphasizes that the PCs explain, successively, as much aspossible of tr(Σ), but the current property shows, intuitively, that theyalso do a good job of explaining the off-diagonal elements of Σ. Thisisparticularly true when the PCs are derived from a correlation matrix, andis less valid when the covariance matrix is used and the variances of theelements of x are widely different (see Section 2.3).It is clear from (2.1.10) that the covariance (or correlation) matrix canbe constructed exactly, given the coefficients and variances of the first rPCs, where r is the rank of the covariance matrix. Ten Berge and Kiers(1999) discuss conditions under which the correlation matrix can be exactlyreconstructed from the coefficients and variances of the first q (< r)PCs.A corollary of the spectral decomposition of Σ concerns the conditionaldistribution of x, given the first q PCs, z q , q =1, 2,...,(p − 1). It canbe shown that the linear combination of x that has maximum variance,conditional on z q , is precisely the (q + 1)th PC. To see this, we use theresult that the conditional covariance matrix of x, givenz q ,isΣ − Σ xz Σ −1zz Σ zx ,where Σ zz is the covariance matrix for z q , Σ xz is the (p × q) matrixwhose (j, k)th element is the covariance between x j and z k ,andΣ zx isthe transpose of Σ xz (Mardia et al., 1979, Theorem 3.2.4).It is seen in Section 2.3 that the kth column of Σ xz is λ k α k . The matrixΣ −1zz is diagonal, with kth diagonal element λ −1k, so it follows thatq∑Σ xz Σ −1zz Σ zx = λ k α k λ −1kλ kα ′ k=k=1q∑λ k α k α ′ k,k=1