MACHINE LEARNING TECHNIQUES - LASA

75
The kernel k provides a metric of similarity across datapoints. Using k may allow extracting
features common to all training datapoints. These features are some non-linear correlations not
visible in the original space.
Classical kernels one finds in the literature are:
• Homogeneous Polynomial Kernels: ( )
k x, x' = x, x' , p∈•
;
• Inhomogeneous Polynomial Kernels: ( ) ( p
)
p
k x, x' = x, x' + c , p∈•
, c≥0;
• Hyperbolic Tangent Kernel (similar to the sigmoid function):
( ) ( θ x x )
k x, x' = tanh + , ' , θ∈°
;
• Gaussian Kernel (translation-invariant): ( )
The most popular kernel is the Gaussian kernel.
x−x'
−
k x x e σ σ
2
2
, ' = , ∈° .
5.2 Which kernel, when?
A recurrent question we often hear in class is “how does one choose the kernel”. A number of
recent works have addressed this problem by offering techniques to learn the kernel. This often
amounts to building esimators that rely on mixture of kernels. One then learns how to combine
these kernels to find the optimal mixture. There is however unfortunately no good recipe to
determine which kernel to use when. One may either try different kernel in an iterative manner
and look at which provides the best result for a given technique. For instance, if you perform
classification, you may want to compare the performance obtained on both training and testing
sets after crossvalidation when doing so with either Gaussian kernel or Polynomial kernel. You
may then pick the kernel that yields to best performance.
While choosing the kernel is already an issue, once a kernel is chosen, one is left with the
problem of chosing the hyperparameter of the kernel. These are, for instance, the variance σ in
the Gaussian kernel or the order of the polynome p in the polynomial kernels. There again,
there is no good recipe. When using a Gaussian kernel, a number of approaches, known under
the term of kernel polarization, have been proposed whereby one learns the optimal covariance
parameters. Most of these approaches however are iterative by nature and rely on a discrete
sampling of values taken by the parameters, using some heuristics or boosting techniques to
optimize the search through these parameter values.
© A.G.Billard 2004 – Last Update March 2011