MACHINE LEARNING TECHNIQUES - LASA

115
5.9.3 Curse of dimensionality, choice of hyperparameters
The above showed that when considering particular condition on the form of the kernel matrix for
GP, GPR and GMR become equivalent. In its generic form, however, GP is not equivalent to
GMM and offers a more powerful tool for inference.
Unfortunately, this is done at the expense of being prohibitive in its computation steps. Indeed,
Gaussian Process is a very expensive method as inference grows with the number of datapoints
3
with O( M )
. Advances in the field investigate sparsifying techniques to decrease in a clever
manner the number of training points. Unfortunately, most methods are based on some heuristics
to determine which points are deemed better than others. As a result the gain of doing a full GP
inference with heuristic-driven sparsification over GMM is no longer obvious.
Another drawback lies in the choice of the kernel function and in particular of the kernel width.
This is illustrated in Figure 5-18. Too small a kernel width may lead to poor generalization as it
allows solely points very close to one another to influence inference. On the other hand too large
a kernel width may smooth out local irregularities. In this respect, GMM is a more powerful tool to
provide generalization outside areas not covered by the datapoints, while still encapsulating the
local non-linearities. I
Figure 5-18: Effect of the width of a gaussian kernel on a classification task (SVM, top) and regression
(GPR, bottom). When the kernel width (gamma) is very small, the generalization becomes very poor (i.e. the
system overfits the data and is unable to estimate correctly on samples that have never been seen).
Choosing appropriate parameters for the kernel depends on the data and is one of the main challenges in
kernel methods.
[DEMOS\CLASSIFICATION\SVM-KERNEL-WIDTH.ML] [DEMOS\REGRESSION\GPR-KERNEL-WIDTH.ML]
© A.G.Billard 2004 – Last Update March 2011