MACHINE LEARNING TECHNIQUES - LASA

93
( ) ( , )
f x = sign w x + b
(5.34)
We can now define a learning algorithm for linearly separable problems. First, observe that
among all hyperplanes separating the data, there exists a unique optimal hyperplane,
distinguished by the maximum margin of separation between any training point and the
hyperplane, defined by:
{ }
i
maximizew ∈H,
b∈Rmin x−x , x∈ H, w, x + b= 0, i = 1,..., M (5.35)
While in the simple classification problem we had presented earlier on, it was sufficient to simply
compute the distance between the two cluster’ means to define the normal vector and so the
hyperplane, here, the problem of finding the normal vector that leads to the largest margin is
slightly more complex.
To construct the optimal hyperplane, we have to solve for the objective function τ ( w)
:
subject to the inequality constraints:
minimize
1 2
∈ ∈R
τ ( w)
= w
(5.36)
2
w H,
b
i
( )
i
y w, x + b ≥1, ∀ i=1,...M
(5.37)
Consider the points for which the equality in (5.37) holds (requiring that there exists such a point
is equivalent to choosing a scale for w and b). These points lie on two hyperplanes
i
( + = ) and H2 ( wx , 1)
i b
H1 wx , b 1
+ =− with normal w and perpendicular distance from
the origin 1 − b / w . Hence d = d = 1/ w and the margin is simply 2 / w . Note that
1
2
+ −
H and
H are parallel (they have the same normal) and that no training points fall between them. Thus
we can find the pair of hyperplanes which gives the maximum margin by minimizing
to constraints (5.37) that ensures that the class label for a given
for y = −1.
i
2
w , subject
i
i
x will be + 1 if y = + 1, and − 1
Let us now rephrase the minimization under constraint problem given by (5.36) and (5.37) in
terms of the Lagrange multipliers α
i, i= 1,..., l, one for each of the inequality constraints in (5.37).
Recall that the rule is that for constraints of the form ci
≥ 0, the constraint equations are
multiplied by positive Lagrange multipliers and subtracted from the objective function, in (5.36), to
form the Lagrangian. For equality constraints, the Lagrange multipliers are unconstrained. This
gives the Lagrangian:
We must now minimize
of
L with respect to all the
P
l
1 2
i
i
( ) ( )
L wb , , α ≡ w − α y wx , + b + α .
P i i
2 i= 1 i=
1
l
∑ ∑ (5.38)
L with respect to
P
w , b , and simultaneously require that the derivatives
α vanish, all subject to the constraints α ≥ 0. This is a convex
i
i
© A.G.Billard 2004 – Last Update March 2011