MACHINE LEARNING TECHNIQUES - LASA

86
1
This implies that det ( A − ) is a constant (and is equal to ± 1 in the particular case of whitenend
mixtures).
Assuming then that the data is zero-mean and white, that the number of sources equal the
dimension of the data, i.e. N , and replacing in (5.20), one can see that maximizing the
negentropy is equivalent to minimizing mutual information, as the two quantities differ only by a
constant factor, i.e.
N
I s ,... s cst. J x .
( ) ( )
1
N
= −∑ (5.23)
Given that the mutual information of a random vector is nonnegative, and zero if and only if the
components of the vector are independent, minimization of mutual information for ICA is
interesting in that it has a single global minimum. Unfortunately, the mutual information is difficult
to approximate and optimize on the basis of a finite sample of datapoints, and much research on
ICA has focused on alternative contrast functions. The earliest approach that optimizes for the
negentropy using the fast-ICA algorithm, see Section 2.3.6, relies on an adhoc set of non-linear
functions. While not very generic, this approach remains advantageous in that it allows estimating
each component separately and in that it does the estimation only on uncorrelated patterns.
i=
1
i
Non-linear Case
We will here assume that the sources and the original datapoints have the same dimension and
these are zero-mean and white. In practice this can be achieved by performing first PCA and then
reducing to the smallest set of eigenvectors with best representation. Hence, given a set of
j
j=
observation vectors { } 1,...
i
M
X = x , we will look for the sources s 1
,.... N
i=
1,... N
s , such that:
s and
1
i
= Wxi
W = A − . As in linear ICA, we do not compute A and then its inverse, but rather
estimate directly its inverse by computing the columns of W.
While the fast-ICA method in linear ICA relied on chosing one or two predefined non-linear
functions for the transformation, Kernel ICA solves ICA by considering not just a single non-linear
{ 1
.,
2
.,...}
function, but an entire function space H:= h ( ) h ( )
of candidate nonlinearities. The hope
is that using a function space makes it possible to adapt to a variety of sources and thus would
h, h ,...
make the algorithms more robust to varying source distributions. Each function
1 2
from
goes
° N → ° . We can hence determine a set of N transformations
j=
M
j
{ } 1...
i
h : x H, i 1,... N
i
→ = with associated kernels ( ) ( )
K = h ., h . , i= 1,... N. Beware
i i i
that we are here comparing the projections of the datapoints along the dimensions i= 1,... N of
the original dataset, which is thought to be the same dimension as that of the sources. Each
matrix K is then M × M .
i
In Section 5.4, when considering kernel CCA, we saw that one can determine the projections that
h, h ,...
maximize the correlation across the variables using different sets of projections 1 2
measured by ( K K )
1 ,.... N
. This is
ρ . This quantity is comprised between zero and one. While CCA
aimed at finding the maximal correlation, kernel ICA will try to find the projections that maximize
© A.G.Billard 2004 – Last Update March 2011