Notes on Canonical Correlation

Version: September 30, 2010
<strong>Notes</strong> on Canonical Correlation
Suppose we have a collection of random variables in a (q + p) × 1
vector X that we partition in the following form (and supposing
without loss of generality that p ≤ q):
where
X =
⎛
⎜
⎝
X 1
.
X p
− − −
X p+1
.
X p+q
µ =
⎞
⎟
⎠
=
⎛ ⎞
⎜ µ 1 ⎟
⎝
µ 2
⎛
⎜
⎝
X 1
− − −
X 2
⎠ ; Σ =
and remembering that Σ 21 = Σ ′ 12, and
Suppose
⎛
⎜
⎝
⎞
⎟
⎠
∼ MVN(µ, Σ) ,
Σ 11 Σ 12
Σ 21
⎞
⎟
⎠ ,
Σ 22
Cor(a ′ X 1 , b ′ X 2 ) = a ′ Σ 12 b/ √ a ′ Σ 11 a √ b ′ Σ 22 b .
Σ −1
11 Σ 12 Σ −1
22 Σ ′ 12a = λa ,
with roots λ 1 ≥ λ 2 ≥ · · · ≥ λ p ≥ 0, and corresponding eigenvectors
a 1 , . . . , a p . Also, let
Σ −1
22 Σ ′ 12Σ −1
11 Σ 12 b = λb ,
1

with roots λ 1 ≥ λ 2 ≥ · · · ≥ λ p ≥ 0 and λ p+1 = λ q = 0; the
corresponding eigenvectors are b 1 , . . . , b p .
Looking at the two linear combinations, a ′ iX 1 (called the i th canonical
variate in the first set), and b ′ iX 2 (called the i th canonical variate
in the second set), the squared correlation between them is λ i ; the i th
canonical correlation is √ λ i . The maximum correlation between any
two linear combinations is √ λ 1 , and is obtained for a 1 and b 1 . For
a i and b i , these are uncorrelated with every canonical variate up to
that point, and maximize the correlation subject to that restriction.
Points to make:
a) The matrices Σ −1
11 Σ 12 Σ −1
22 Σ ′ 12 and Σ −1
22 Σ ′ 12Σ −1
11 Σ 12 are not
symmetric and so the standard eigenvector/eigenvalue decompositions
are not straightforward. However, the two matrices
and
are symmetric. Also,
and
Σ −1/2
11 Σ 12 Σ −1
22 Σ ′ 12Σ −1/2
11
Σ −1/2
22 Σ ′ 12Σ −1
11 Σ 12 Σ −1/2
22
Σ −1/2
11 Σ 12 Σ −1
22 Σ ′ 12Σ −1/2
11 e i = λ i e i ,
Σ −1/2
22 Σ ′ 12Σ −1
11 Σ 12 Σ −1/2
22 f i = λ i f i ,
where the roots, i.e., the λ i s, are the same as before. We can then
obtain a i = Σ −1/2
11 e i , and b i = Σ −1/2
22 f i . Both Σ −1/2
11 and Σ −1/2
22 are
constructed from the spectral decompositions of Σ 11 = PDP ′ and
Σ 22 = QFQ ′ as Σ −1/2
11 = PD −1/2 P ′ and Σ −1/2
22 = QF −1/2 Q ′ . Note
2

the normalizations of Var(a ′ iX 1 ) = a ′ iΣ 11 a ′ i = e ′ iΣ −1/2
11 Σ 11 Σ −1/2
11 e i =
1 and Var(b ′ iX 2 ) = 1.
b) There are three different normalizations that are commonly
used for a i and b i :
(i) leave as unit length so a ′ ia i = b ′ ib i = 1;
(ii) make the largest value 1.0 in both a i and b i ;
(iii) do as we did above and make a ′ iΣ 11 a ′ i = 1 = b ′ iΣ 22 b ′ i.
(c) Special cases: When p = 1 and q = 1, λ 1 is the (simple)
squared correlation between two variables; when p = 1 and q > 1,
λ 1 is a squared multiple correlation. In considering a ′ iX 1 versus X 2 ,
λ i is the squared multiple correlation of a ′ iX 1 with X 2 ; b i gives the
regression weights.
(d) When moving to the sample, all items have direct analogues.
The one restriction on sample size is n ≥ p + q + 1.
(e) Suppose the variables X 1 and X 2 are transformed by nonsingular
matrices, A p×p and B q×q , as follows:
Y 1 = A p×p X 1 + c p×1
Y 2 = B q×q X 2 + d q×1
The same canonical variates and correlations using Y 1 and Y 2 would
be generated as from X 1 and X 2 ; the weights in a i and b i would be
on the transformed variables, obviously. In particular, we could work
with standardized variables without loss of any generality, and just
use the correlation matrix.
3

(f) To evaluate H 0 : Σ 12 = 0, a likelihood ratio test is available:
∏
−(n − 1 − (1/2)(p + q + 1)) ln p (1 − λ i ) ∼ χ 2 pq .
Also, sometimes a sequential process is used to test the remaining
roots until nonsignificance is reached:
−(n − 1 − (1/2)(p + q + 1)) ln
p ∏
i=k+1
i=1
(1 − λ i ) ∼ χ 2 (p−k)(q−k) .
This latter sequential procedure is a little problematic because there
is no real control over the overall significance level with this strategy.
Generally, there is some tortuous difficulty in interpreting the
canonical weights substantively. I might suggest using a constrained
least-squares approach (iteratively moving from one set to a second),
where the weights are forced to be nonnegative.
4