MACHINE LEARNING TECHNIQUES - LASA

54
( ) ( ) ( ) ( ) ( ) ( )
{ α 0 ,..., α 0 , 0 ,..., 0 , 0 ,...,
0
K
µ µ
K K }
Θ = ∑ ∑
0 1 1 1
E-step:
At each step t, estimate, for each Gaussian k, the probability that this Gaussian is being
responsible for generating each point of the dataset by computing:
i k ( t) k( t)
( t)
( )
p
( )
( | ,
t
k
x µ ∑ ) ⋅α
t
k
αk = p( k|
Θ ) =
i k ( t)
t
∑ pk
x | j,
µ ⋅α
(3.18)
j
j
( )
( )
M-step:
Recompute the means, Covariances and prior probabilities so as to maximize the log-likelihood of
the current estimate:
probabilities
( t)
( | )
p k Θ :
( t)
( L( X)
)
log |
Θ and using current estimate of the
∑
( )
j ( t)
p( k| x , Θ )
j ( t)
p k| x , Θ ⋅
k( t 1)
j
µ + =
∑
j
x
j
(3.19)
( + 1)
kt
∑ =
∑
j
( )
( )
( | , )
+ 1 ( + 1)
( µ )( µ )
i ( t)
p( k| x , Θ )
p k x Θ x − x −
( t 1)
α +
k
j t j k t j k t
=
∑
i
∑
j
j ( t)
( | , Θ )
Pk x
M
T
(3.20)
(3.21)
Note finally that we have not discussed how to choose the optimal number of states K. There are
various techniques based on determining a tradeoff between increasing the number of states and
hence the number of parameters (increasing greatly the computation required to estimate such a
large set of parameters) and the improvement that such an increase brings to the computation of
the likelihood. Such a tradeoff can be measured by either of the BIC, DIC or AIC criteria. A more
detailed description is given in 7.2.3.
Figure 3-16 shows an example of clustering using a 3 GMM model with full covariance matrix and
illustrates how a poor initialization can result in poor clustering.
© A.G.Billard 2004 – Last Update March 2011