MACHINE LEARNING TECHNIQUES - LASA

96
Figure 5-7: Linear separating hyperplanes for the non-separable case.
5.7.2 Support Vector Machine for Non-linearly Separable Datasets
The above algorithm for separable data, when applied to non-separable data, will find no feasible
solution: this will be evidenced by the objective function (i.e. the dual Lagrangian) growing
arbitrarily large. In order to handle non-separable data, one must relax the constraints (5.37). This
can be done by introducing positive slack variables ξ
i, i= 1,..., M , in the constraints, which then
become:
T i
w x + b≥+ 1− ξ for y =+ 1
(5.47)
i
i
T i
w x + b≤− 1+ ξ for y =− 1
(5.48)
ξ ≥ 0 i
Thus, for an error to occur the corresponding ξ
i
must exceed unity, so
i
i
i
∀ (5.49)
∑
ξ i i
is an upper bound
on the number of training errors. Hence a natural way to assign an extra cost for errors is to
change the objective function to be minimized to include a cost function, such as
M
⎛⎛ 2 C ⎞⎞
min { ⎜⎜ w + ∑ ξi
,
w, ξ M
⎟⎟
⎝⎝
i=
1 ⎠⎠
whereC is a parameter to be chosen by the user, a larger C corresponding to assigning a higher
penalty to errors. As it stands, this is a convex programming problem and the Wolfe dual problem
becomes:
subject to:
The solution is again given by:
1
i j i
L α ≡ α − αα yy x,
x
j
D ∑ i ∑ (5.50)
i j
i 2 i,
j
max { ( )
α
C
≤ ≤ (5.51)
M
i
α y = 0
0 αi
∑ i
(5.52)
i
Ns
w = ∑ α
(5.53)
i=
1
i i
i
yx
© A.G.Billard 2004 – Last Update March 2011