MACHINE LEARNING TECHNIQUES - LASA

53
Theα are the so-called mixing coefficients. Their sum is 1, i.e.
k
K
∑
k = 1
α = 1
k
. These coefficients are
usually estimated together with the estimation of the parameters of the Gaussians (i.e. the means
and covariance matrices). In some cases, they can however be set to a constant. For instance to
give equal weight to each Gaussian (in this case, α 1
k
= ∀ k = 1,... K). When the
K
coefficients are estimated, they end up representing a measure of the proportion of data points
that belong most to that particular Gaussian (this is similar to the definition of theα seen in the
case of Mixture of Gaussians in the previous section). In a probabilistic sense, these coefficients
represent the prior probability with which each Gaussian may have generated the whole dataset
M
j
and can hence be written αk
= p( k) =∑ p( k|
x)
.
j=
1
Learning of GMM requires determining the means and covariance matrices and prior probabilities
of the K Gaussians. The most popular method relies on Expectation-Maximization (E-M). We
advise the reader to take a detour here and read the tutorial by Gilmes provided in the annexes of
these lectures notes for a full derivation of GMM parameters estimation through E-M. We briefly
summarize the principle next.
We want to maximize the likelihood of the model’s parameters
1 1 1
{ α ,..... α K , µ ,..... µ
K , ,..... K ,}
Θ= ∑ ∑ given the data, that is:
Assuming that the set of M datapoints { }
j=
1
(iid), we get:
( ) ( )
max L Θ | X = max p X | Θ (3.15)
Θ
X
Θ
j
M
= x is identically and independently distributed
M K
j k k
( Θ ) = ∑α k
⋅p( x µ ∑ )
max p X | max | ,
Θ
Θ
∏ (3.16)
j=
1 k=
1
Taking the log of the likelihood is often a good approach as it simplifies the computation. Using
the fact that the optimum *
log f x , one can
compute:
x of a function f ( x)
is also an optimum of ( )
( Θ ) = p( X Θ)
max p X | max log |
Θ
Θ
⎛⎛
max log | , max log | ,
Θ
Θ
j=
1 k= 1 j= 1 ⎝⎝ k=
1
M K M K
j k k j k k
∏∑α k
⋅p( x µ ∑ ) = ∑ ⎜⎜∑α k
⋅p( x µ ∑ )
⎞⎞
⎟⎟
⎠⎠
(3.17)
The log of a sum is difficult to compute and one is led to proceed iteratively by calculating an
approximation at each step (EM), see Sections 9.4.2.2 The final update procedure runs as
follows:
Initialization of the parameters:
Initialize all parameters to a value to start with. The
p 1
,..,
K
p can for instance be initialized with a
uniform prior, while the means can be initialized by running K-Means first. The complete set of
parameters is then given by:
© A.G.Billard 2004 – Last Update March 2011