Notes on Principal Component Analysis

More documents

Recommendations

Info

$Producing Beautiful Documents With TEX and LATEX$

components that have variance of λ i ≥ 1 — otherwise, the component would “explain” less variance than would a single variable. A major rub — if I do principal components on the correlation matrix, P, and on the original variance-covariance matrix, Σ, the structures obtained are generally different. This is one reason the “true believers” might prefer a factor analysis model over a PCA because the former holds out some hope for an invariance (to scaling). Generally, it seems more reasonable to always use the population correlation matrix, P; the units of the original variables become irrelevant, and it is much easier to interpret the principal components through their coefficients. The j th principal component is a ′ jX: Cov(a ′ jX, X i ) = Cov(a ′ jX, b ′ X), where b ′ = [0 · · · 0 1 0 · · · 0], with the 1 in the i th position, = a ′ jΣb = b ′ Σa j = b ′ λ j a j = λ j times the i th component of a j = λ j a ij . Thus, Cor(a ′ jX, X i ) = λ j a ij √ λj σ i = √ λj a ij σ i , where σ i is the standard deviation of X i . This correlation is called the loading of X i on the j th component. Generally, these correlations can be used to see the contribution of each variable to each of the principal components. If the population covariance matrix, Σ, is replaced by the sample covariance matrix, S, we obtain sample principal components; if the population correlation matrix, P, is replaced by the sample correlation matrix, R, we again obtain sample principal components. These structures are generally different. 4
The covariance matrix S (or Σ) can be represented by ⎡ √ ⎤ ⎡ √ ⎤ ⎡ λ1 · · · 0 λ1 · · · 0 a ′ 1 S = [a 1 , . . . , a p ] ⎢ . . ⎣ √ ⎥ ⎢ . . ⎦ ⎣ √ ⎥ ⎢ . ⎦ ⎣ 0 · · · λp 0 · · · λp a ′ p or as the sum of p, p × p matrices, ⎤ ⎥ ⎦ ≡ LL ′ S = λ 1 a 1 a ′ 1 + · · · + λ p a p a ′ p . Given the ordering of the eigenvalues as λ 1 ≥ λ 2 ≥ · · · ≥ λ p ≥ 0, the least-squares approximation to S of rank r is λ 1 a 1 a ′ 1 + · · · + λ 1 a r a ′ r, and the residual matrix, S − λ 1 a 1 a ′ 1 − · · · − λ 1 a r a ′ r, is λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p. Note that for an arbitrary matrix, B p×q , the Tr(BB ′ ) = sum of squares of the entries in B. Also, for two matrices, B and C, if both of the products BC and CB can be taken, then Tr(BC) is equal to Tr(CB). Using these two results, the least-squares criterion value can be given as Tr([λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p][λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p] ′ ) = ∑ k≥r+1 λ 2 k . This measure is one of how bad the rank r approximation might be (i.e., the proportion of unexplained sum-of-squares when put over ∑p k=1 λ 2 k). For a geometric interpretation of principal components, suppose we have two variables, X 1 and X 2 , that are centered at their respective means (i.e., the means of the scores on X 1 and X 2 are zero). In 5
Page 1 and 2: Version: August 4, 2011 Not
Page 3: 2) a i and a j are orthogonal, i.e.
Page 7 and 8: Let a 11 and a 21 denote the weight
Page 9 and 10: symmetry, or is an equicorrelation
Page 11 and 12: 0.2 Little Jiffy Chester Harris nam
Page 13 and 14: etween the subject’s row vector a

Notes on Principal Component Analysis

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?