MACHINE LEARNING TECHNIQUES - LASA

49
3.1.4 Clustering with Mixtures of Gaussians
An extension of the soft K-means algorithm consists of fitting the data with a Mixture of
Gaussians (not to be confused with Gaussian Mixture Model (GMM))) which we will review later
on. Instead of simply attaching a responsibility factor to each cluster, one attaches a density of
probability measuring how well each cluster represents the distribution of the data. The method is
bound to converge to a state that maximizes the likelihood of each point to belong to each
distribution.
Soft-Clustering methods are part of model-based approaches to clustering. In clustering with
mixture of Gaussians, the model is naturally a Gaussian. Other model-based methods use, for
instance, the Poisson or the Normal distributions.
The main advantages of model-based clustering are:
• It can make use of well-studied statistical inference techniques;
• Its flexibility in choosing the component distribution;
• It obtains a density estimation for each cluster;
• It is a “soft” means of classification.
Clusters with mixtures of Gaussian places K distributions, whose barycentres are located on the
cluster means, as in Figure 3-13.
Figure 3-12: Examples of clustering with Mixtures of Gaussians (the grey circles represent the first and
second variances of the distributions). [DEMOS\CLUSTERING\GMM-CLUSTERING-SIMPLE.ML]
Algorithm
Assignment Step (E-step): The responsibilities are
r
k
i
=
∑
k '
α
k
α
( 2πσ
k )
k '
1
1
N
( 2πσ
k ' )
e
N
⎛⎛ 1
⎜⎜−
⋅d
⎜⎜ 2
⎝⎝ σ k
e
⎛⎛ 1
⎜⎜−
⋅d
⎜⎜ 2
⎝⎝ σ k '
( µ , x )
k
i
⎞⎞
⎟⎟
⎟⎟
⎠⎠
( µ , x )
k'
i
⎞⎞
⎟⎟
⎟⎟
⎠⎠
(3.6)
© A.G.Billard 2004 – Last Update March 2011