MACHINE LEARNING TECHNIQUES - LASA

84
5.5 Kernel ICA
Adapted from Kernel Independent Component Analysis, F. Bach and M.I. Jordan, Journal of Machine Learning
Research 3 (2002) 1-48
In Section 2.3, we covered the linear version of Independent Component Analysis. We first here
revisit ICA and extend this to a non-linear transformation through kernel ICA. To this end, we first
show that ICA can also be solved by minimimization of mutual information and extend this
observation to the non-linear case.
Linear ICA
i=
i
ICA assumes that the set of M observations { } 1...
independent sources { 1 q}
j
M
X = x was generated by a set of statistically
j=
1,... N
S = s ,.... s , with q ≤ N through a linear transformation A :
q
A:
° → °
N
s → x=As
where A: is an unknown N×
q mixing matrix.
ICA consists then in estimating both A and S knowing only X .
ICA bears important similarity with Probabilistic PCA, see Section 2.1.5. The sources S are latent
random variables, i.e. each source s ,.... 1
s was generated by an independent random process and
q
hence as an associated distribution p . ICA differs from PPCA in that it requires that the sources
s i
be statistically independent. PCA in contrast requires solely that the projections be uncorrelated.
Statistical independence is a stronger constraint than un-correlatedness, see definitions in
Annexes, Sections 9.2.7 and 9.2.8.
To solve for ICA would require doing a multi-dimensional density estimation to estimate each of
the densities p ,.... .
s
p This is impractical and hence ICA is usually solved by approximation
1 s q
techniques. In Section 2.3, we saw one method to solve ICA using a measure of non-gaussianity.
The intuition behind this approach was that the distribution of the mixture of independent sources
becomes closer to a Gaussian distribution than the distribution of each source independently.
This, of course, assumed that the distribution of each source is non-gaussian. This idea is
illustrated in Figure 5-3. A measure of non-gaussianity was proposed based on the Negentropy
J y H y H y
H y of the
( ) = ( Gauss ) − ( ). The negentropy measures by how much the entropy ( )
y s of the distribution of the sources differs from the entropy H( y )
current estimate ~
Gaussian distribution with the same mean and covariance as that of the distribution of y . In
information theory, the entropy is a measure of the uncertainty attached to the information
contained in the observation of a given variable. The more entropy, the more uncertain the event
is, see Section 9.3. The notion of entropy can be extended to joint and conditional distributions.
When observing two variables, the joint entropy is a measure of the information conveyed by the
observation of one variable onto the other variable. It is hence tightly linked to the notion of
information. Unsurprisingly, ICA can hence also be formulated in terms of mutual information, as
we will see next.
Gauss
of a
© A.G.Billard 2004 – Last Update March 2011