MACHINE LEARNING TECHNIQUES - LASA

92
Let us now rewrite this result in terms of the input patterns
dot product. We get the decision function:
⎛⎛ 1 1
⎞⎞
sgn , i
i
y= ⎜⎜ x x − x,
x + b
⎜⎜
⎟⎟
m+ i= | yi=+ 1
m
⎟⎟
− i= | yi
⎝⎝
=+ 1 ⎠⎠
⎛⎛ 1 1
⎞⎞
i
i
= sgn ⎜⎜ k( x, x ) − k( x,
x ) + b
⎜⎜
⎟⎟
m+ i= | yi=+ 1
m
⎟⎟
− i= | yi
⎝⎝
=+ 1 ⎠⎠
x i , using the kernel k to compute the
∑ ∑ (5.31)
∑ ∑ (5.32)
If b = 0 , i.e. the two classes' means are equidistant to the origin, then k can be viewed as a
probability density when one of its arguments is fixed. By this, we mean that it is positive and has
unit integral,
∫
( ʹ′ )
k x , x dx= 1 ∀xʹ′∈X
X
In this case, y takes the form of the so-called Bayes classifier separating the two classes, subject
to the assumption that the two classes of patterns were generated by sampling from two
probability distributions that are correctly estimated by the Parzen Windows estimator of the two
class densities,
and
where x∈ X .
p
p
+
−
1
i
: = ∑ k x, x
m
+ iy | i =+ 1
− iy | i =−1
( )
1
i
: = ∑ k x, x
m
( )
Thus, given some point x , the class label is computed by checking which of the two values p +
or
p −
is larger. This is the best decision one can take if one has no prior information on the data
distribution.
5.7.1 Support Vector Machine for Linearly Separable Datasets
SVM consists of determining a hyperplane that determines a decision boundary in a binary
classification problem. We consider first the linear case and then move to the non-linear case.
Linear Support Vector Machines
Let us assume for a moment that our datapoints x live in a feature space H . The class of
hyperplanes in the dot product space H is given by:
with the corresponding decision functions
wx , + b= 0, where w∈H,
b∈° (5.33)
© A.G.Billard 2004 – Last Update March 2011