MACHINE LEARNING TECHNIQUES - LASA

82
One can further add normalization constraints on the denominator, i.e.
α K α = α K α = 1.
2 2
x x x y y y
Then, as done for linear CCA, see Section 2.2, one can express this optimization as a
generalized eigenvalue problem of the form:
2
0 KK
x y α ⎛⎛
x
K
x
0 ⎞⎞ αx
⎛⎛ ⎞⎞⎛⎛ ⎞⎞ ⎛⎛ ⎞⎞
⎜⎜ = ρ
⎜⎜
⎟⎟⎜⎜ 2
KK
y x
0 ⎟⎟ ⎟⎟ ⎜⎜ ⎟⎟
α ⎜⎜
⎜⎜ ⎟⎟
y 0 K ⎟⎟
⎝⎝ ⎠⎠⎝⎝ ⎠⎠ α
⎝⎝ y ⎠⎠⎝⎝ y⎠⎠
(5.17)
A first difficulty that arises when solving the above problem is the fact that its solution is usually
always ρ = 1, irrespective of the kernel. This is due to the fact that the intersection between the
spaces spanned by the columns of K and K is usually non-zero 4 . In this case, there exist two
x
αx
αysuch that Kxαx Kyαy
vectors and
y
= , that are solution of (5.17). Finding a solution to
(5.17) will hence yield different projections depending on the kernel, but all of these will have
maximal correlation. This hence cannot serve as a means to determine which non-linear
transformation K is most appropriate.
A second difficulty when solving (5.17) is that it requires inverting the Gram matrices. These may
not always be invertible, especially as these are not full rank (because of the constraint of zero
mean in feature space).
To counter this effect, one may use a regularization parameter (also called ridge parameter) κ
on the norm of the transformation yielded by K , K and end up with the following regularized
problem:
x
2
⎛⎛⎛⎛
Mκ
⎞⎞
⎞⎞
⎜⎜ Kx
+ I 0 ⎟⎟
⎛⎛ 0 KK
x y⎞⎞⎛⎛αx ⎞⎞
⎜⎜ ⎟⎟
2
αx
ρ
⎜⎜⎝⎝
⎠⎠
⎟⎟⎛⎛ ⎞⎞
⎜⎜ =
⎜⎜
⎟⎟⎜⎜
2
KK
y x
0 ⎟⎟ ⎟⎟
α ⎜⎜ ⎟⎟⎜⎜ ⎟⎟
⎝⎝ y
y
{
Mκ
α
1442443⎠⎠⎝⎝ ⎠⎠ ⎜⎜ ⎛⎛ ⎞⎞ ⎝⎝ ⎠⎠
0 K
y
+ I ⎟⎟
A α ⎜⎜
⎜⎜ ⎟⎟
2
⎟⎟
⎝⎝144444424444443
⎠⎠ ⎠⎠
⇔ Aα
= ρBα
y
B
(5.18)
Thanks to the regularization term, when M is very large, the right-handside matrix becomes PSD
(which is usually the case as the number of datapoints M>>N, the dimension of the dataset; if this
is not the case, then one makes κ very large). In this case, the matrix B can be decomposed
T
into B= C C. Replacing, this yields a classical eigenvalue problem of the form:
T
−
( ) 1 −1
C AC β = ρβ, with β = Cα.
Note that the solution to this eigenvalue problem neither shows the geometry of the kernel
canonical vectors nor gives an optimal correlation of the variates. On the other hand, the
4
K and
x
K are centered and are hence at most of dimensions M − 1 . The dimension of the
y
intersection between the spaces spanned by the column vectors of these matrices is
dim ( C( K ) , ( )) dim ( ( )) dim( ( ) )
x
C Ky ≥ C Kx + C Ky
− M . If M>2, i.e. if we have more than two
datapoints, then the space spanned by the two canonical basis intersect.
© A.G.Billard 2004 – Last Update March 2011