MACHINE LEARNING TECHNIQUES - LASA

31
2.3.2 Why Gaussian variables are forbidden
A fundamental restriction of ICA is that the independent components must be non-Gaussian for
ICA to be possible. To see why Gaussian variables make ICA impossible, assume that the mixing
matrix is orthogonal and the s i are Gaussian. Then x 1 and x 2 are Gaussian, uncorrelated, and of
unit variance. Their joint density is given by:
( , )
p x x
1 2
1
e
2π
⎛⎛ x1+
x2
⎞⎞
⎜⎜−
2
⎟⎟
⎝⎝ ⎠⎠
= (2.21)
This distribution is illustrated in Figure 2-10. The Figure shows that the density is completely
symmetric. Therefore, it does not contain any information on the directions of the columns of the
mixing matrix A. This is why A cannot be estimated.
Figure 2-10: The multivariate distribution of two independent Gaussian variables
More rigorously, one can prove that the distribution of any orthogonal transformation of the
Gaussian (x 1 ,x 2 ) has exactly the same distribution as (x 1 ,x 2 ), and that x 1 and x 2 are independent.
Thus, in the case of Gaussian variables, we can only estimate the ICA model up to an orthogonal
transformation. In other words, the matrix A is not identifiable for Gaussian independent
components. Note that if just one of the independent components is Gaussian, the ICA model can
still be estimated.
2.3.3 Definition of ICA
Let x= { x1
,..., x N
} be a N-dimensional random vector of observables.
ICA consists of finding a linear transform s
are linearly independent.
To proceed, one builds a general linear model of the form:
= Wx so that the projections s ,..., 1 q
s of x through W
© A.G.Billard 2004 – Last Update March 2011