MACHINE LEARNING TECHNIQUES - LASA

52
The estimation of Mixture of Gaussians we reviewed in the previous section assumed a diagonal
covariance matrix for each Gaussian. This is rather restrictive as it forces the Gaussians to have
their axes aligned with the original axes of the data, see Figure 3-14. As a result, they are illsuited
for estimating highly non-linear densities that have local correlations.
Figure 3-14: A spherical Mixture of Gaussian Model can only fit Gaussians whose axes are
aligned with the data axes. It may require far more local models to account for local correlations
across the dimensions of the data X.
Figure 3-15: In contrast, a Gaussian Mixture Model can exploit local correlations and adapt the
covariance matrix of each Gaussian so that it aligns with the local direction of correlation.
Gaussian Mixture Model is a generic method that fits a mixture of Gaussians with full covariance
matrices. Each Gaussian serves as a local linear model and in effect performs a local PCA on the
data. By mixing the effect of all these local models, one can obtain a very complex density
estimate, see as an illustration the schematics in Figure 3-14 and Figure 3-15.
j
j=
Let suppose that we have at our disposal { } 1,...
X
i
M
= x the set of M N-dimensional data points
i=
1,... N
whose density we wish to estimate. We consider that these are observed instances of the
variable x . The probability distribution function or density of x is further assumed to be
k
k
composed of a mixture of k = 1,.., K Gaussians with mean ∑ , i.e.
µ and covariance matrix
K
k k k k k k
( ) = α
k
⋅
k( | µ , ∑ ), with
k( | µ , ∑ ) = ( µ , ∑ )
∑ (3.14)
p x p x p x N
k=
1
© A.G.Billard 2004 – Last Update March 2011