MACHINE LEARNING TECHNIQUES - LASA

104
Examples:
Figure 5-12 illustrates the application of SVR on the same dataset (these data were generated
using MLdemos). When using a linear kernel with a large e, one can completely encapsulate the
datapoints, while still yielding a very poor fit. Using a Gaussian kernel allows to fit better the nonlinearities.
This is however very sensitive to the choice of penalty given to poor fit through
parameter C. A high penalty will result in a smoother fit.
Figure 5-12: Example of e-SVR on a two-dimensional datasets. Datapoints are shown in plain dot. The
regression signal is in solid line. The boundaries around the e-insensitite tube are drawn in light grey lines.
using a linear kernel (top) and a Gaussian kernel (bottom). TOP: effect of the increasing the e-tube with
e=0.02 (left) and e=0.1 (right). BOTTOM: effect of increasing parameter C (from left to right, C=10, 100 and
1000) that penalies for poorly fit datapoints.
SVR as all other regression techniques remain very sensitive also the choice of hyperparameter
for the kernel. This is illustrated in Figure 5-13. Too small a kernel width will lead to overfitting of
the data. Too large may lead to very poor performance. The latter effect can be compensated by
choosing appropriate value for the hyperparameters of the optimization function, namely C and e,
see Figure 5-14. The small kernel width that had led to a poor fit in Figure 5-13 is compensated
by relaxing the constraints using a large e and smaller C. We also see that reducing the
constraints and widening e decreases the number of support vectors. While the very tight fit in
Figure 5-13 used almost all datapoints as support vector, the looser fit in Figure 5-14 used much
less.
© A.G.Billard 2004 – Last Update March 2011