MACHINE LEARNING TECHNIQUES - LASA

27
The CCA algorithm consists thus in finding the optimum of ρ under the above two constraints.
This again can be resolved through Lagrange and gives:
C C C w
−1 2
xy yy yx x xx x
= λ C w
(2.18)
with λ the Lagrange multiplier of the constraint.
This is a generalized eigenproblem of the form Ax = λBx
. If C is invertible (which is the case if
xx
all the columns of X are independent), then the above problem reduces to a single symmetric
−
eigenvector problem of the form B 1 Ax= λx.
2.2.1 CCA for more than two variables
Given sets of K multivariate random variables
k
k
X with dimension , 1,...,
M× n k = K(note that
k
while each variate can have different dimensions n , k = 1,..., K , they must all have the same
number of observed instances M). The generalized CCA problem consists then in determining the
= , with p= min k = 1.. K
n k
, such that:
k k
W w
=
set { i }
i 1...
p
min
W
∑
XW
l≠ k, k, l=
1,..., K F
k
( )
k k l l
T
k
s.t. W C W = I, ∀ k = 1,.., K,
kk
− XW
k l
wiCklw j
= 0, ∀ i, j = 1,..., p, i ≠ j.
(2.19)
Where is the Frobenius norm (Euclidean distance for matrices). The above consists of K
F
minimization under constraint problems. Unfolding the objective function, we have the sum of the
squared Euclidean distances between all of the pairs of the column vectors of the
k k
matrices XW, k= 1,..., K . This problem can be solved by using singular value decomposition
for arbitrary K.
2.2.2 Limitations
CCA is dependent on the coordinate system in which the variables are described, so even if there
is a very strong linear relationship between two sets of multidimensional variables, depending on
the coordinate system used, this relationship might not be visible as a correlation. Kernel CCA
has been proposed as an extension to CCA, where data are first projected into a higher
dimensional representation. KCCA is covered in Section 5.4 of these Lecture Notes.
© A.G.Billard 2004 – Last Update March 2011