MACHINE LEARNING TECHNIQUES - LASA

32
x= A⋅ s+
ε
Where A is a q
× N ‘mixing’ matrix, s ,..., 1
s
q
the set of independent components and ε a q-
dimensional random noise vector 1 . The independent components are latent variables, meaning
that they cannot be directly observed.
This formulation reduces the ICA problem to ordinary estimation of a latent variable model.
Because the estimation of the latent variable in the noisy case can be very tricky, the majority of
ICA research has concentrated on the noise-free ICA model, where:
x
x
= A⋅ s
(2.22)
q
= ∑ a s
(2.23)
i ij j
j=
1
W = A −1 (this is possible only if A is invertible; to this
The matrix W is then the inverse of A, i.e.
end, one can ensure that A is full rank by reducing its dimension to that of the independent
component, see below)..
Hypotheses of ICA:
The starting point for ICA is the following assumptions:
• Without loss of generality, we can assume that, both the mixture variables x and the
independent components s have zero mean. Observe that the observable variables x
can always be centered by subtracting the sample mean, i.e. x x'
E{ x'
}
= − .
Consequently, the independent component have also zero mean,
−
E s = A 1 E x .
since { } { }
• The components s i are statistically independent. Statistical independence is rigorously
defined in Section 9.2.7. It is a stronger constraint than uncorrelatedness (which is
ensured through PCA). Hence, ICA decomposition results usually in different estimates
from that found through PCA decomposition.
• As discussed before, we must also assume that the independent components have non-
Gaussian distributions. Usually, assuming that the data follow a Gaussian distribution is
handy and is done in many other techniques we will see in this course (see e.g. PPCA or
GMM). Gaussian distributions are so-called parametric distribution that is the distribution
is fully determined once its parameters have been defined. Hence, assuming a Gaussian
distribution simplifies the estimation of the density as one must solely estimates the
parameters of the Gaussian (or mixture of Gaussians, as in GMM). Since ICA does not
make any assumption regarding the form of the distribution of the independent
component, it looks as if ICA would estimate the full density of s. This is a very difficult
problem. ICA will overcome this difficulty by not estimating explicitly the density of s.
Rather; the density of s can be recovered by sampling through x and using the inverse
transformation.
1 Note the similarity between this model and that introduced in PPCA, see Section 2.1.5; the
difference here lies in the optimization method whereby ICA optimizes for statistical
independence and PCA optimizes for maximal variance.
© A.G.Billard 2004 – Last Update March 2011