MACHINE LEARNING TECHNIQUES - LASA

69
Figure 4-2: Illustration of the fact that in a probabilistic regressive model the uncertainty on the predictive
model increases quadratically with the amplifude of the input.
It also grows with the number of training points. This effect can be diminished if one uses a
sparse representation of the data. Parametric models, such as Gaussian Mixture Regression
which we will see next, can also remove the requirement to store all of the training points. Here
we considered solely linear regressive models, non-linear probabilistic regressive models, such
as Gaussian Process Regression, will be covered in Section 5.8.
4.4 Gaussian Mixture Regression
Adapted from Hsi Guang Sung, Gaussian Mixture Regression and Classification, PhD thesis, Rice University,
2004.
In Section 3.1.5, we introduced Gaussian Mixture Models (GMM). Assume
N
z ∈° a multi-
p z = p z ,..., 1
z of the joint distribution of x
N
= 1,.., K Gaussians ( | , ) ( , )
k k
k k
k N
pk
z µ ∑ = N ⎡⎡
⎣⎣µ
∑ ⎤⎤ with mean
⎦⎦ µ ∈°
k N×
N
∑ ∈° , i.e.
dimensional variable GMM builds a model of ( ) ( )
using a mixture of k
and covariance matrix
\
( )
1
p z p z z z
k= 1 k=
1 k
2π
∑
2
K
K
1
k k k
T
k
−
k
( ) = ∑α k
⋅ k ( | µ , ∑ ) = ∑ α
k
⋅ exp
N ( −µ ) ( ∑ ) ( −µ
) (4.18)
The mixing coefficientsα satisfy
k ∑ αk
= 1.
K
k = 1
Gaussian Mixture Regression (GMR) exploits the joint density to construct its estimate using the
expectation on the conditional of the variable onto which we wish to make a prediction. If for
instance, we wish to predict z1
from knowing all other entries z z , then we can compute:
2
,... N
{ ( N)
} ( N)
∫
(4.19)
zˆ = E p z | z ,..., z = z ⋅p z | z ,..., z ⋅dz
1 1 2 1 1 2 1
© A.G.Billard 2004 – Last Update March 2011