MACHINE LEARNING TECHNIQUES - LASA

94
quadratic programming problem, since the objective function is itself convex, and those points
which satisfy the constraints also form a convex set (any linear constraint defines a convex set,
and a set of N simultaneous linear constraints defines the intersection of N convex sets, which
is also a convex set). This means that we can equivalently solve the following dual problem:
maximize L , subject to the constraints that the gradient of L with respect to w and b vanish,
P
α ≥ .
and subject also to the constraints that the 0
i
Requesting that the gradient of
L with respect to
P
w and b vanish
P
∂
i i
P j i
, j=1,....N
j
w L = w −∑
∂
α y x
(5.39)
j
i
gives the conditions:
∂
∂b L
P
∑ i
(5.40)
i
i
=− α y = 0
M
w = ∑ α
(5.41)
M
i=
1
i=
1
i i
i
yx
i
∑ α i
y = 0.
(5.42)
Since these are equality constraints in the dual formulation, we can substitute them into (5.38) to
give:
1
j
L α = α y − αα y y x , x .
( )
∑ ∑ (5.43)
i i j i
D i i j
i 2 i,
j
Note that we have now given the Lagrangian different labels (P for primal, D for dual) to
emphasize that the two formulations are different: L and L arise from the same objective
function but with diferent constraints; and the solution is found either by minimizing
maximizing
L . Note also that if we formulate the problem with b = 0
D
P
D
L or by
P
, which amounts to
requiring that all hyper-planes contain the origin (this is a mild restriction for high dimensional
spaces, since it amounts to reducing the number of degrees of freedom by one), support vector
training (for the separable, linear case) then amounts to maximizing L with respect to the α ,
subject to constraints (5.42) and positivity of theα , with solution given by (5.41). Notice that
there is a Lagrange multiplier α for every training point. In the solution those points for which
i
α
i
> 0 are called support vectors and lie on one of the hyperplanes H1,
2
α = , and lie either on
1
H , or on that side of
1
points have
i
0
inequality in (5.37) is satisfied.
H or
2
i
D
H . All other training
H or H such that the strict
2
i
For these machines, the support vectors are the critical elements of the training set. They lie
closest to the decision boundary; if all other training points were removed (or moved around, but
so as not to cross H or
1
H ), and training was repeated, the same separating hyperplane would
2
be found.
© A.G.Billard 2004 – Last Update March 2011