MACHINE LEARNING TECHNIQUES - LASA

106
5.8.1 n-SVR
The e parameter in e-SVR determines the desired accuracy of the approximation. Determining a
good e may be difficult in practice. n-SVR is an alternative method that suppresses the need to
pre-specifying the free parameter C in (5.65), by introducing (yet another!) parameter n. The idea
if that this new parameter will make it easier to estimate e as one finds a tradeoff between model
complexity and slack variables.
By introducing a penalty throughν ≥ 0 , the optimization of (5.65) becomes:
( )
( * M
1 2 ⎛⎛ 1
w ξ ε) w ⎜⎜νε + ∑( ξ *
i
+ ξi
)
minimize L , , + C
2 ⎝⎝
i
( )
i
( )
⎧⎧ i
w,
φ x + b− y ≤ ε + ξi
⎪⎪
⎪⎪ i
*
subject to ⎨⎨y − w,
φ x −b≤ ε + ξi
⎪⎪
*
⎪⎪ξi
≥ 0, ξi
≥ 0, ε ≥ 0
⎩⎩
M
i=
1
⎞⎞
⎟⎟
⎠⎠
(5.74)
Notice that we now optimize also for the value of e. The term νε in the objective function is now
a weighting term that balances a growth ofε and the effect this has on the increase of the poorly
fit datapoints (last term of the objective function).
This last new equality constraint on e yields a new Lagrange solution with associated Lagrange
multiplier β . One can then proceed to the same steps as done when solving e-SVM, i.e. write the
Lagrangian, take the partial derivatives, write the dual and the solution to KKT conditions (see
Scholkopf & Smola 2002 for details). This yields the following n-SVR optimization problem:
For ν ≥ 0, C>0
⎛⎛
M
M
* 1
⎞⎞
i * *
i j
max { ⎜⎜∑( αi −αi
) y − ∑( αi −αi
)( α
j
−α
j) k( x , x ) ⎟⎟,
⎝⎝ i= 1 2 i, j=
1
⎠⎠
(*)
M
α ∈°
M
*
∑ ( αi
αi
)
subject to − = 0,
i, j=
1
α
(*)
i
M
*
∑ ( i i )
i, j=
1
⎡⎡ C ⎤⎤
∈ ⎢⎢ 0, ,
M ⎥⎥
⎣⎣ ⎦⎦
α + α ≤Cν.
(5.75)
The regression estimate then takes the same form as before and is expressed as a linear
(*)
combination of the kernel estimated on each of the data point with non zero α (the support
i
vectors), i.e.:
M
*
i
( ) ( αi
αi
) ( )
f x = ∑ − k x , x + b.
(5.76)
i, j=
1
As we did before solyly for, we can now also find b and ε by solving the KKT conditions.
© A.G.Billard 2004 – Last Update March 2011