MACHINE LEARNING TECHNIQUES - LASA

111
One can then use the above expression of the joint distribution over f and f* to compute the
posterior distribution of f* given the training and testing sets X, X * and our prior on f
: N 0, K X,
X which yields:
( ( ))
( ) ( )
−1
f*| X*, X, f : N( K X*, X K X, X f,
−1
( ) − ( ) ( ) ( )
K X*, X* K X*, X K X, X K X, X* )
(5.83)
One can then simply sample from this posterior distribution by evaluating the mean and
covariance matrix from (5.83) and generating samples as done previously for the prior distribution
on f.
Figure 5-17 shows three examples of such sampling. In all three plots the shaded area
represents the pointwise mean plus and plus/minus the standard deviation for each input value
(corresponding to the ~97% confidence region), for the posterior distribution. We used the square
exponential covariance function given by ( )
1
k x, x' = exp ⎛⎛
⎜⎜− x−x'
⎝⎝ 2
2
⎞⎞
⎟⎟
⎠⎠
. We plot in light grey the
area around the regression signal that corresponds to +/- one standard deviation (using the
covariance given by (5.83)). This gives a measure of the uncertainty of the inference of the
model. Areas with large uncertainty are due to lack of training points covering that space. From
left to right, we see the effect of adding one new point (in red) in areas where previously there
were no training points on decreasing locally the variance.
Figure 5-17: The confidence of a Gaussian Process is dependent on the amount of data present in a specific
region of space (left). Regions of low data density have lower confidence. By adding points in those regions,
the confidence increases (center), but the regression function will change to adapt to the new data (right).
[DEMOS\REGRESSION\GPR-CONFIDENCE.ML]
The previous model assumed that the process f was noise free. However, when modeling real data, it is
usual to assume some noise superposition. As in the case of the linear regression mode, we can assume
that the noisy version of our non-linear regression model follows:
( )
( σ)
y= f x + ε
ε ~ N 0,
Where the noise ε follows a zero mean Gaussian distribution.
(5.84)
© A.G.Billard 2004 – Last Update March 2011