Notes on Principal Component Analysis

The covariance matrix S (or Σ) can be represented by
⎡ √ ⎤ ⎡ √ ⎤ ⎡
λ1 · · · 0
λ1 · · · 0
a ′ 1
S = [a 1 , . . . , a p ]
⎢
. .
⎣
√
⎥ ⎢
. .
⎦ ⎣
√
⎥ ⎢
.
⎦ ⎣
0 · · · λp 0 · · · λp a ′ p
or as the sum of p, p × p matrices,
⎤
⎥
⎦
≡ LL ′
S = λ 1 a 1 a ′ 1 + · · · + λ p a p a ′ p .
Given the ordering of the eigenvalues as λ 1 ≥ λ 2 ≥ · · · ≥ λ p ≥ 0,
the least-squares approximation to S of rank r is λ 1 a 1 a ′ 1 + · · · +
λ 1 a r a ′ r, and the residual matrix, S − λ 1 a 1 a ′ 1 − · · · − λ 1 a r a ′ r, is
λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p.
Note that for an arbitrary matrix, B p×q , the Tr(BB ′ ) = sum of
squares of the entries in B. Also, for two matrices, B and C, if both
of the products BC and CB can be taken, then Tr(BC) is equal
to Tr(CB). Using these two results, the least-squares criterion value
can be given as
Tr([λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p][λ r+1 a r+1 a ′ r+1 + · · · + λ p a p a ′ p] ′ ) =
∑
k≥r+1
λ 2 k .
This measure is one of how bad the rank r approximation might be
(i.e., the proportion of unexplained sum-of-squares when put over
∑p
k=1 λ 2 k).
For a geometric interpretation of principal components, suppose
we have two variables, X 1 and X 2 , that are centered at their respective
means (i.e., the means of the scores on X 1 and X 2 are zero). In
5