MACHINE LEARNING TECHNIQUES - LASA

68
⎛⎛ 1 T
T
T ⎞⎞ ⎛⎛ 1 T −1
⎞⎞
p( w| X, y) ∝exp⎜⎜− ( y−X w) ( y−X w)
⎟⎟⋅exp⎜⎜− w ∑w
w⎟⎟
⎝⎝ 2 ⎠⎠ ⎝⎝ 2 ⎠⎠
⎛⎛ 1 ⎛⎛ 1 ⎞⎞
∝exp⎜⎜− − ⎜⎜ +∑
2
w ⎟⎟ −
⎝⎝ 2 ⎝⎝σ
⎠⎠
⎛⎛ 1
T −1
⎞⎞
∝exp⎜⎜− ( w−v) ∑v
( w−v)
⎟⎟
⎝⎝ 2
⎠⎠
T
T −1
( w v) XX ( w v)
T
−
T
( w ) and
v ( w )
−2 −2 −1 1
−2 −1
−1
where v = σ σ XX +∑ Xy ∑ = σ XX +∑
⎞⎞
⎟⎟
⎠⎠
(4.15)
The posterior distribution of the weight is thus a Gaussian distribution with mean v and
covariance matrix ∑
ν
. Notice that the first term on the right handside of both terms is the
covariance of the input modulated by the variance of the noise. In a deterministic model where
the variance of the weight is zero and the variance of the noise is the identity, we recover the
classical linear regressive model.
We can compute our best estimate of the weights by computing the expectation over the
posterior distribution, i.e.:
−2 −2 T −1
−
{ ( )} σ ( σ ) 1
w
E p w| y, x = XX +∑ Xy.
(4.16)
This is called the maximum a posteriori (MAP) estimate of w.
Notice that the posterior of the weight and by extension its MAP estimate depend not only on the
input variable X but also on the output variable y . In practice, computation of MAP grows
quadratically with the number of examples M and is a major drawback of such methods. Current
efforts in research related to the development of such regression techniques are all devoted to
reducing the dimension of the training set to make this computation bearable. These are so-called
sparse methods. This is particularly crucial when one extends this computation to kernel
methods, see Chapter 5.
Once the parameters w have been estimated, we can use our probabilistic regressive model to
*
make predictions given new data points. Hence, given a so-called query point x (such point is
usually a point not used for training the model; in other words, this would be a point belonging to
y
* = f x
* . This can be
the testing set or the validation set), we can compute an estimate of ( )
obtained by averaging over all possible values for the weight, i.e.:
where
A
−1 1 T −1
XX
2
w
= +Σ .
σ
( * | * , , ) = ∫ ( * | * , ) ( | , )
p y x X y p y x w p w X y dw
⎛⎛ 1
⎝⎝σ
* T −1 * T −1 *
= N⎜⎜ x A Xy,
x A x
2
⎟⎟
⎞⎞
⎠⎠
(4.17)
The predictive distribution is again Gaussian. Notice that the uncertainty (variance) on the
predictive model grows quadratically with the amplitude of the query point and with the variance
of the weight distribution. This is expected from linear regressive model and hence is a limitation.
This effect is illustrated in Figure 4-2.
© A.G.Billard 2004 – Last Update March 2011