MACHINE LEARNING TECHNIQUES - LASA

74
X
M
i
= x is the set of our M observations, we can construct a projection of X onto the
If { }
i=
1
feature space H through a non-linear mapping φ :
φ : X → H
x→φ
( x)
.
(5.1)
Note that if the dimension of X is N , then the dimension of the feature space may be greater
than N .
Finding the appropriate non-linear transformationφ is difficult in practice. The kernel trick relates
i
to the observation that most methods rely on computing an inner product ,
x x
i
observations ,
j
x x
j
across pairs of
∈ X (e.g., recall that in order to perform PCA, we needed to compute the
T
XX ;
covariance matrix on X , which is given by the dot product on all the data points, i.e.
similarly when performing linear regression, we had to multiply by the inverse of the covariance
on X).
Hence, determining an expression for this inner product in feature space may save us from
having to compute explicitly the mappingφ . This inner product in feature space is expressed
through the kernel function (or simply the kernel) k (.,.).
All the kernel methods we will see in these lecture notes assume that the feature space H into
which the data are sent through the non-linear functionφ is a Reproducing Kernel Hilbert Space
{ 1
.,
2
.,...}
(RKHS). H is hence a space H:= h ( ) h ( )
( ) ( )
of functions from
N
° to ° . In this space,
φ ., x i , φ ., x j ∀x i , x j ∈X
defines a set of functions indexed by x . The “reproducing
property" of the kernel ensures that taking the inner product between a pair of functions in H
k ., x : = φ ., x yields the following kernel:
yields a new function in H . Hence setting ( ) ( )
( i , j ) φ( ., i ), φ( ., j ) φ( , i ), φ( ,
j
)
∞
k x x = x x = ∫ z x z x dz
(5.2)
The kernel in (5.2) is the dot product in feature space. Most techniques covered in these lecture
notes will be based on this kernel.
i
Note that, in the literature, people tend to omit the open parameter on φ (.,
x )
simply φ ( x
i
) . We will follow this notation in the remainder of this document.
−∞
and write
For proper use in the algorithms we will see next, the kernel must satisfy a number of properties
that follow the Mercer’s theorem. Among these, we will retain that k is a symmetric continuous
positive function that maps:
k:
X × X →°
( i , j ) → φ( i ),
φ( j
)
k x x x x
(5.3)
© A.G.Billard 2004 – Last Update March 2011