MACHINE LEARNING TECHNIQUES - LASA

66
4.3 Probabilistic Regression
Probabilistic regression is a statistical approach to classical linear regression and assumes that
the observed instances of x and y have been generated by an underlying probabilistic process of
which it will try to build an estimate. The rationale goes as follows:
If one knows the joint distribution ( , )
p x y of x and y , then an estimate y ) of the output y can be
built from knowing the input x by computing the expectation of y given x ,
{ ( )}
yˆ E p y|
x
= (4.6)
Note that in many cases, if one is solely interested in constructing a regressive model, one does
not need to build a model of the joint density but needs solely to estimate the
conditional ( | )
p y x .
Probabilistic regression extends the concept of linear regression by assuming that the observed
values of y differ from f ( x ) by an additive random noiseε (noise is usually assumed to be
independent of the observable x ):
( , )
y= f x w + ε
(4.7)
Usually, to simplify computation, one further assumes that the noise follows a zero-mean
2
Gaussian distribution with uncorrelated isotropic varianceσ . The covariance matrix is diagonal
with all elements equal to 2
2
ε = N 0, σ . Such an assumption is called
putting a prior distribution over the noise.
σ ; so we write simply ( )
Let us first consider the probabilistic solution to the linear regression problem described before,
that is:
2
( 0, )
T
y xw N σ
= + (4.8)
We have now one more variable to estimate, namely the variance of the noiseσ .
Assuming that all pairs of observables are i.i.d (identically independently distributed), we can
construct an estimate of the conditional probability of y given x and a choice of parameters
w,
σ as:
M
i
( | , , σ) p( y| x , w,
σ)
p y x w
= ∏ (4.9)
i=
1
Solving for the fact that sole the noise model is probabilistic and that it follows a Gaussian
distribution with zero mean, we obtain:
© A.G.Billard 2004 – Last Update March 2011