MACHINE LEARNING TECHNIQUES - LASA

102
M
∂ L =
*
i
w − ∑ ( αi
− αi
) φ( x ) = 0;
∂w
i=
1
(5.68)
∂L
C (*) (*)
= − 0
i( *)
i i
ξ M α − η =
∂
(5.69)
Substituting in (5.66) yields the dual optimization under constraint problem:
M
⎧⎧ 1 * *
i j
⎪⎪ − ∑ ( αi − αi
)( α
j − α
j) ⋅ k( x , x )
⎪⎪ 2 i, j=
1
max { ⎨⎨
M
M
*
αα , ⎪⎪ −
* i *
ε ( αi
+ αi
) + y ( αi
+ αi
⎪⎪ ∑ ∑ )
⎩⎩ i= 1 i=
1
M
* * i ⎡⎡ C ⎤⎤
subject to ∑( αi
− αi
) = 0 and αi
, αi
∈ ⎢⎢ 0,
i=
1
M ⎥⎥
⎣⎣ ⎦⎦
(5.70)
Solving for the above optimization problem will yield solutions for the Lagrange multipliers α * , α .
These can then be used to construct our estimate of the best projection vector w by replacing in
(5.68):
M
w= −
i=
1
*
i
( αi
αi
) φ( x )
∑ (5.71)
To determine the value of the offset b , one must further solve for the Karush-Kuhn-Tucker (KKT)
conditions:
i
i
( i
y w ( x ) b)
i
i
y w ( x ) b
α ε + ξ + − , φ − = 0,
i
(
i
)
α ε + ξ − + , φ + = 0,
* *
i
and
⎛⎛
⎜⎜
⎝⎝
⎛⎛
⎜⎜
⎝⎝
C
M
C
M
⎞⎞
− αi
⎟⎟ξi
= 0,
⎠⎠
* ⎞⎞ *
− αi
⎟⎟ξi
= 0.
⎠⎠
(5.72)
The above optimization problem can be solved using interior-point optimization, see (Scholkopf &
Smola, 2002).
It is worth spending a little bit of time looking at the geometrical implication of conditions (5.72).
(*) C
The last two conditions show that, when the Lagrange multipliers take valueα i
= , the
M
corresponding slack variable (*)
can take arbitrary value and hence the corresponding points can
ξ
i
be anywhere, including outside the e-insensitive tube. Vice-versa, when
have ξ (*) i
= 0 . These become the support vectors.
(*) ⎤⎤ C ⎡⎡
αi
∈ ⎥⎥ 0,
M ⎢⎢
⎦⎦ ⎣⎣ , we
© A.G.Billard 2004 – Last Update March 2011