MACHINE LEARNING TECHNIQUES - LASA

90
5.7 Support Vector Machines
Adapted from "Learning with Kernels" by B. Scholkopf and A. Smola, MIT Press 2002, and "A Tutorial on Support
Vector Machines for Pattern Recognition", by C.J.C. Burges, Data Mining and Knowledge Discovery, 2, 1998.
Support vector machine (SVM) is probably one of the most popular applications of kernel
methods. SVM exploits the kernel trick to extend classical linear classification to non-linear
classification. It has been shown to be very powerful at separating highly intertwined data. Its
simplicity of use and the large number of available software makes it easy to implement. We will
here very briefly review the principle and the derivation of the algorithm. We will highlight on the
sensivity to some hyperparameter so as to guide the potential user and ensure optimal use of the
algorithm.
Linear Case
Let us first start by considering a very simple classification problem which illustrates well the
reasoning behind using Kernel methods for classification. Suppose we are given two classes of
objects. We are then faced with a new object, and we have to assign it to one of the two classes.
This problem can be formalized as follows:
Consider a training set composed of M input-output pairs where each input { } i =
i
1...
associated with a label { } i =
i
1...
y
M
. The label
In SVM, we consider solely binary classification problems, i.e.:
i
y denotes the class to which the pattern
{ , i=
1... M
i i
x y } X { 1 }
x
M N
∈ ° is
i
x belongs.
∈ × ± (5.27)
Note that there exist extensions to multiclass SVM. We will here focus first on the binary
classification case.
Given this training set, we wish to build a model of the relationships across the input points and
their associated class label that would be a good predictor of the class to which each pattern
belongs and would allow us to do inference: that is, given a new pattern x , we could estimate the
class to which this new pattern belongs. In some sense, for a given new pattern x , we would
choose a corresponding y, so that the pair { xyis , } somewhat similar to the training examples. To
this end, we need a notion of similarity in X and in{ ± 1}
.
Similarity Measures: Characterizing the similarity of the outputs { 1}
± is easy. In binary
classification, only two situations occur: two labels can either be identical or different. The choice
of the similarity measure for the inputs, on the other hand, is more complex and is tighly linked to
k x, x'
gives a
the idea of kernel. As we have seen previously in these lecture notes, the kernel ( )
measure of similarity across two datapoints x and x' . A natural choice for the kernel when
considering the simple linear classification problem outlined above is to take the dot product, i.e.:
N
k x, x' = x, x' =∑ x x'
(5.28)
( )
The geometrical interpretation of the canonical dot product is that it computes the cosine of the
angle between the vectors x and x ' , provided they are normalized with length 1.
i=
1
i
i
© A.G.Billard 2004 – Last Update March 2011