MACHINE LEARNING TECHNIQUES - LASA

100
In SVM, we found that minimizing w was interesting as it allowed a better separation of the two
classes. Scholkopf and Smola argue that, in SVR, minimizing w is also interesting albeit for a
different geometrical reason. They start by computing the e-margin given by:
{ }
( ) ( ) ( ) ( ) ( )
mε f : = inf φ x −φ x' , ∀x, x', s.t. f x − f x' ≥ 2ε
(5.61)
Figure 5-11: Minimizing the e-margin in SVR is equivalent to maximizing the slope of the function f.
Similarly to the margin in SVM, the e-margin is a function of the projection vector w . As illustrated
in Figure 5-11, the flatter the slope w of the function f, the larger the margin. Conversely, the
steeper the slope is, the larger the width of the e-insensitive tube. Hence to maximize the margin,
we must minimize w (minimizing each projection of w will flatten the slope). The linear illustration
in Figure 5-11 holds in feature space as we will proceed to a linear fit in feature space.
Finding the optimal estimate of f can now be formulated as an optimization problem of the form:
⎛⎛1
min ⎜⎜
w ⎝⎝ 2
2 ε
{ w + C⋅R [ f]
⎞⎞
⎟⎟
⎠⎠
(5.62)
ε
Where R [ f]
is a regularized risk function that gives a measure of the e-insensitive error:
1 M i i
[ ] = − ( )
ε
R f ∑ y f x
(5.63)
M
ε
i=
1
C in (5.62) is a constant and hence a free parameter that determines a tradeoff between
minimizing the increase in the error and optimizing for the complexity of the fit. This procedure is
called e-SVR.
Formalism:
© A.G.Billard 2004 – Last Update March 2011