MACHINE LEARNING TECHNIQUES - LASA

110
defined by its mean m( x)
and covariance function ( , ')
x’ that span the data space):
( ) =Ε ( )
m x f x
k x x (k is defined for each pair of point x,
{ }
( )( ( ) ( ))
{ }
( , ') ( ) ( )
k x x =Ε f x −m x f x −m x
(5.79)
For simplicity, most GP assume a zero-mean process, i.e. m( x ) = 0. While the above
description may be multidimensional, most regression techniques on GP assume that the output
f ( x)
is unidimensional. This is a limitation in the application of the process for regression as it
allows making inference solely on a single dimension, say y= f ( x),
y∈° . For multidimensional
inference, one may run one GP per output variable.
Using (5.79) and (5.78) and assuming zero-mean distribution, the baysian regression model can
be rewritten as a Gaussian process defined by:
T
{ ( )} φ ( ) { }
E f x = x E w = 0,
T T
T
{ } φ( ) { } φ( ) φ( ) wφ( )
( ) ( ) ( )
k x, x' = E f x f x' = x E ww x' = x Σ x'
(5.80)
We are now endowed with a probabilistic representation of our real process f. The value taken by
f at any given pair of input x, x’ is jointly Gaussian with zero mean and covariance given
by k( x, x ')
. This means that the estimate of f can be drawn only from looking conjointly at the
distribution of f across two or more input variables. In practice, to visualize the process, one may
sample a set *
estimates of f, such that:
Where ( *, *)
* M *
i
X of M* of data points X* { x }
K X X is a M * M *
i j
using ( )
ij ( )
= and compute f* a M*-dimensional vector of
i=
1
( ( ))
f* N 0, K X*, X*
: (5.81)
× covariance matrx, whose elements are computed
K X*, X * = k x , x , ∀ i, j = 1.... M *. Note that if M*
> D, i.e. the number of
datapoints exceed the dimension of the feature space, the matrix is singular as the rank of K is D.
Generating such a vector is called drawing from the prior distribution of f as it uses solely
information on the query datapoints themselves and the prior assumption that the underlying
process if jointly Gaussian and zero mean, as given by (5.81). A better inference can be made if
one can make use of prior information in terms of a set of training points. Consider the set
X
M
i
{ x }
i=
1
= as the training datapoints, one can then express the joint distribution of the
estimates f and f* associated with the training and testing points respectively as:
( , ) ( *, )
( ) ( )
⎡⎡f
⎤⎤ ⎛⎛ ⎡⎡K X X K X X ⎤⎤⎞⎞
⎢⎢ N 0,
f *
⎥⎥ ⎜⎜ ⎢⎢
⎥⎥⎟⎟
⎣⎣ ⎦⎦ ⎜⎜ ⎢⎢K X, X * K X*, X * ⎥⎥⎟⎟
⎝⎝ ⎣⎣
⎦⎦⎠⎠
: (5.82)
© A.G.Billard 2004 – Last Update March 2011