MACHINE LEARNING TECHNIQUES - LASA

73
5 Kernel Methods
These lecture notes have sofar covered linear methods for performing a variety of computation,
ranging from dimensionality reduction, clustering and classification to regression. The term linear
refers to the assumptions that all of these transformations could be expressed as linear
transformation of the form y= Ax (where y and xare given and A is the unknown
transformation).
Kernel methods relax the assumption of a linear transformation, so as to perform non-linear
regression, classification, etc. Kernel methods proceed by first projecting the data through a nonlinear
transformation into a feature space and then perform the same type of computation (e.g.
classification or regression), as in the linear case, in the feature space. This principle is illustrated
in Figure 5-1.
Figure 5-1: Illustration of the principle of non-linear classification through Support Vector Machine. The data
are first projected from input space to a feature space via a non-linear map φ . Linear separation is then
performed in the feature space. Figure from "Learning with Kernels" by B. Scholkopf and A. Smola, MIT
Press 2002.
In these lecture notes, we will cover solely a subset of the existing Kernel methods, focusing on
kernel methods that extend the algorithms see previously for dimensionality reduction (PCA, ICA,
CCA to kernel PCA, kernel ICA, kernel CCA), for clustering (from K-Means to kernel K-means),
for classification (from linear classifier to support vector machine) and for regression (from
probabilistic regression to Gaussian Process Regression).
5.1 The kernel trick
All kernel methods we will see here are based on the so-called kernel trick. The kernel trick stems
from a variety of fundamental mathematical principles which we will skip here, focusing solely on
explaining its basic principle.
© A.G.Billard 2004 – Last Update March 2011