MACHINE LEARNING TECHNIQUES - LASA

117
The weight w determines the slope of the sigmoid function. This is an open parameter which
should be chosen so as to ensure optimal classification, or conversely minimal penalty due
misclassification. We now have all the tools to proceed to Gaussian Process classification.
As in Gaussian Process Regression, we can start by putting a Gaussian prior on the distribution
of the parameter w , i.e. w~ N( 0, Σ
w ) and compute the log of the posterior distribution on the
weight given our dataset X = { x 1 ,.... x
M
} with associated labels { 1 ,..., M
Y y y }
= :
M
1 T
1
log p( w| X, Y)
=− w Σ
ww+
log
i
2
i=
1
( x )
1+
e −
∑ T
(5.93)
Unfortunately the posterior does not have an analytical solution, a contrario to the regression
case. One can however observe that the posterior is concave and hence its maximum can be
found using classical optimization method, such as Newton’s methods or conjugate gradient
descent. Figure Figure 5-19 illustrates nicely how the original weight distribution is tilted as an
effect of fitting the data.
w
Figure 5-19: a) Contours of the prior distribution p(w) = N(0, I). (b) dataset, with circles indicating class +1
and crosses denoting class −1. (c) contours of the posterior distribution p(w|D). (d) contours of the predictive
distribution p(y_=+1|x_), adapted from C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for
Machine Learning, the MIT Press, 2006.
© A.G.Billard 2004 – Last Update March 2011