MACHINE LEARNING TECHNIQUES - LASA

78
As in the original space, in feature space, the correlation matrix C φ
can be diagonalized and we
have now to find the eigenvalues λi ≥ 0, i= 1... M,
satisfying:
Cv
i
φ
= λ v
i
i
( ) φ
λi
φ( )
j i j i
⇒ φ x , C v = x , v , ∀ i, j = 1,... M
(5.10)
All solutions v with λ ≠ 0 lie in the span of the
Developping the left hand-side
M
j i j i l
( x ), Cφv = ( x ),
Cφ∑
l ( x )
φ φ α φ
=
M
∑
l=
1
1
=
M
1
=
M
l=
1
( x ),
Cφ
( x )
α φ φ
i j l
l
M
∑
l=
1
( x ),
FF ( x )
α φ φ
i j T l
l
M
( x ), ∑ ( x ) ( x ),
( x )
M
i j j j l
∑αl
φ φ φ φ
l= 1 j=
1
φ
1
M
(x ),…, φ(x )
( )
M
i i i j
i i j
j=
1
and we can thus write:
Cv
φ
= λv = λ∑ α φ x
(5.11)
Replacing the latter expression in the definition of the correlation matrix, one gets:
1
M α φ φ φ φ λ α φ φ
M M M
i j j j l i i j
∑ l ( x ), ∑ ( x ) ( x ), ( x ) =
i∑ j ( x ), ( x ) . (5.12)
l= 1 j= 1 j=
1
Using the kernel trick, one can define the Gram Matrix K , whose elements are composed of the
dot product between each pair of datapoints projected in feature space, i.e.
i j
( ) φ( )
K φ x , x .
ij
= Beware that K is M × M , where M is the number of data points.
We can finally rewrite the expression given in (5.12) as an eigenvalue problem of the form:
2 i
i
K α = Mλi
Kα
, i=
1... M
i
i
Kα
= Mλα
i
(5.13)
This is the dual eigenvalue problem of finding the eigenvectors
i
v of C.
© A.G.Billard 2004 – Last Update March 2011