MACHINE LEARNING TECHNIQUES - LASA

99
5.8 Support Vector Regression
In SVM, we have consider a mapping from the input data X onto a binary output y =± 1. Support
Vector regression (SVR) extends the principle of SVM to allow a mapping f to a real value
output:
f : X → °
( )
x→ y = f x
(5.58)
SVR starts from the standard linear regression model and applies it in feature space. Assume a
X →φ
X ∈ H. Then, SVR seeks a linear mapping in
projection of the data into a feature space ( )
feature space of the form:
( ) φ ( )
f x = w, x + b, w∈H,
b∈° (5.59)
To recall, SVM approximates the classification problem by choosing a subset of data points, the
support vectors, to support the decision function. This sparsification of the training dataset is the
key strength of the algorithm. When considering non-separable datasets, SVM introduced a slack
variable to give room for imprecise classification. SVR proceeds similarly and tries to find the
optimal number of support vector while allowing for imprecise fitting. The allowed imprecision of
the fitting through f is measured by a parameter ε ≥ 0 and is measured through an e -loss
function:
{ ε}
( ) ( )
y− f x = max 0, y− f x − ,
(5.60)
ε
Points with a non-zero e -loss function lie outside the e-insensitive tube that surrounds the
function f , see Figure 5-10.
Figure 5-10: Effect of a non-linear regression through SVR. The tightness of the e-insensitive tube around
the regression signal varies along the state space as an effect of the distance in feature space between the
support vectors (the support vector are plain circles). Datapoints within the e-insensitive tube do not
influence the regression model and are indicated with un-filled circles.
© A.G.Billard 2004 – Last Update March 2011