MACHINE LEARNING TECHNIQUES - LASA

85
Figure 5-3: TOP: Marginal (left) and joint (right) distributions of two statistically independent sources, a
Gaussian distribution and a uniform distribution. BOTTOM: The marginal distributions after whitening are
closer to that of a Gaussian distribution.
ICA by minimization of mutual information
ICA searches statistically independent sources. These sources must therefore have minimal
mutual information. A measure of the mutual information across the q sources is given by:
q
1
( s
1,...
q) = ( i) − ( ) −log det ( )
∑ (5.20)
I s h s h x A −
where h( x)
is the entropy of the distribution of the observations.
i=
1
To recall, ICA started with the assumption that the data was centered and white, i.e. ~ ( 0, )
x N I .
In practice, this requires to first substract the mean of the data and then to proceed to a
decorrelation through PCA, followed by a normalization, see Section 2.3.4. By extension if
x~ N 0, I then the sources are also centered and white and hence:
( )
T − T −
{ } { }( )
I = E ss = A E xx A
(5.21)
1 1 T
Given that ( I )
det =1
, we have:
−1 T −1
T
( A E{ xx }( A ) )
−
T
( ) { }
det =1
⇔
T
( )
−
( ) ( )
1 1
det A det E xx det A =1.
(5.22)
© A.G.Billard 2004 – Last Update March 2011