MACHINE LEARNING TECHNIQUES - LASA

101
The optimization problem given by (5.62) can be rephrased as an optimization under constraint
problem of the form:
1 2
minimize w 2
⎧⎧ i
i
⎪⎪
w,
φ( x ) + b− y ≤ε
subject to ⎨⎨
∀ i=
1,... M
i
i
⎪⎪y − w,
φ( x ) −b≤ε
⎩⎩
(5.64)
Minimizing for the norm of w , we ensure a) that the optimization problem at stake is convex
(hence easier to solve) and b) that the conditions given in (5.64) are easier to satisfy (for a small
value of w , the left handside of each condition is more likely to be bounded above by ε ).
Note that satisfying the conditions of (5.64) may be difficult (or impossible) for an arbitrary small
*
e, one can once more introduce the set of slack variables ξi and ξ
i
( i = 1... M)
for each datapoint,
to denote whether the datapoint is on the left or righthandside of the function f 5 . The slack
variables measure by how much each datapoint is incorrectly fitted. This yields the following
optimization problem:
1 C
minimize w +
2 M
i
( )
i
( )
M
∑( ξi
+ ξi
)
2 *
i=
1
⎧⎧ i
w,
φ x + b− y ≤ ε + ξi
⎪⎪
⎪⎪ i
*
subject to ⎨⎨y − w,
φ x −b≤ ε + ξi
⎪⎪
*
⎪⎪ξi
≥ 0, ξi
≥ 0
⎩⎩
(5.65)
Again, one can solve this quadratic problem through Lagrange. Introducing a set of Lagrange
multipliers α , η ≥ 0 for each of our inequalities constraints and writing the Lagrangian:
i
i
1 C C
L , , *, = +
2 M
M
M
M
2 * * *
( w ξξ b) w ∑( ξi + ξi ) − ∑( ηξ i i
+ ηξ
i i )
M
∑
i=
1
M
i=
1
i
i= 1 i=
1
i
i
( i
y w,
( x ) b)
− α ε + ξ + − φ −
∑
i
i
( y w,
( x ) b
i
)
− α ε + ξ − + φ +
* *
i
(5.66)
Solving for each of the partial derivatives:
∂L ∂L ∂L
= 0; = 0; = 0
(*)
∂b ∂w ∂ξ
M
∂ L =
*
∑ ( αi
− αi
) = 0;
(5.67)
∂b
i=
1
5 We will use the shorthand
ξ
(*)
when the notation applies to both
i
ξ
i
*
and ξ
i
.
© A.G.Billard 2004 – Last Update March 2011