MACHINE LEARNING TECHNIQUES - LASA

33
• In our general definition of the ICA model given previously, we have assumed that A was
a q× N matrix. Here, we will focus on a simplified version of ICA whereby one assumes
that the unknown mixing matrix A is square, i.e. that the number of independent
components is equal to the number of observed mixtures and thus A is. q× q (note that
this assumption can be sometimes relaxed (see extensions proposed in Hyvarinen et al
2001). This can be done for instance by performing first PCA on the original dataset and
then use a reduced set of q dimensions obtained by PCA for ICA. If A is square, then,
after estimating the matrix A , one can compute its inverse, W = A −1 , and obtain the
independent component simply by:
−1
s Wx A x
= = (2.24)
• The data is white, i.e. each datapoint is uncorrelated and the variance of the dataset is
equal to unity. This will be a basic preprocessing step in ICA, as we will discuss next.
2.3.4 Whitening
A useful preprocessing strategy in ICA is to first whiten the observed variables. This means that
before the application of the ICA algorithm (and after centering), we transform the observed
vector x linearly so that we obtain a new vector x%, which is white, i.e. its components are
uncorrelated and its variance equal unity. In other words, the covariance matrix of x%equals the
identity matrix:
E xx %% T = I
(2.25)
{ }
The whitening transformation is always possible. One popular method for whitening is to use the
%% T
T , where U is the orthogonal
eigen-value decomposition of the covariance matrix E{ xx }
= UDU
matrix of eigenvectors of the basis of x and D is the diagonal matrix of its eigenvalues,
= ( ,..., 1 n ). Note that { T
}
D diag λ λ
E xx%% is the empirical means, i.e. it is estimated from the
available data samples. Whitening can now be done by computing:
1
−
2 T
% (2.26)
x = UD U x
The matrix
1
2
D −
is computed by a simple component-wise operation, such that:
1 1 1
− ⎛⎛ − − ⎞⎞
2 2 2
D = diag ⎜⎜d1 ,..., dn
⎟⎟
⎝⎝ ⎠⎠
It is easy to check that now Exx {%% T } = I.
Whitening transforms the mixing matrix into a new one, A % :
.
1
−
2 T
% (2.27)
x= UD U As= As %
The utility of whitening resides in the fact that the new mixing matrix A % is orthogonal. This can be
seen from
© A.G.Billard 2004 – Last Update March 2011