MACHINE LEARNING TECHNIQUES - LASA

36
2.3.6.1 Negentropy
To obtain a measure of non-Gaussianity that is zero for a Gaussian variable and always
nonnegative, one often uses a slightly modified version of the definition of differential entropy,
called the negentropy. The negentropy J is defined as follows
where
y
gauss
( ) ( ) ( )
J y = H y − H y
(2.30)
gauss
is a Gaussian random variable of the same covariance matrix as y to the abovementioned
properties. The negentropy is always non-negative, and it is zero if and only if y has a
Gaussian distribution. Negentropy is invariant for invertible linear transformations.
The advantage of using negentropy, or, equivalently, differential entropy, as a measure of
nongaussianity is that it is well justified by statistical theory. In fact, negentropy is in some sense
the optimal estimator of nongaussianity, as far as statistical properties are concerned. The
problem in using negentropy is, however, that it is computationally very difficult. Estimating
negentropy using the above definition would require an estimate (possibly nonparametric) of the
pdf of the independent component. Therefore, simpler approximations of negentropy are very
useful. For instance, Hyvarinen et al 2001 propose to use the following approximation. For a
given non-linear quadratic function G, then:
( ) ∝⎡⎡
( )
( ) − ( ( Gauss ))
J y
⎣⎣
E G y E G y
T
( ) ∝⎡⎡
( )
( ) − ( Gauss )
2
( )
J y E G w x E G y
⎣⎣
⎤⎤
⎦⎦
⎤⎤
⎦⎦
2
(2.31)
2.3.6.2 FastICA for one unit
To begin with, we shall show the one-unit version of FastICA. By a "unit" we refer to a
computational unit. As we will see in Section 6.7.2, this can also be considered as the output of a
neuron.
The FastICA learning rule determines a direction, i.e. a unit vector w such that the projection
T
wx maximizes nongaussianity. Nongaussianity is here measured by the approximation of the
J w x .
T
empirical measure of Negentropy on the projection, i.e. ( )
T
Recall that the variance of wxmust here be constrained to unity; for whitened data this is
equivalent to constraining the norm of w to be unity.
The FastICA is based on a fixed-point iteration scheme for finding a maximum of the
T
nongaussianity of wx. It can be also derived as an approximative Newton iteration, minimizing
for the derivative of the Negentropy, i.e.
either of the two following quadratic functions:
1
G1
( y) = log ( cosh ( a⋅y)
) , 1 ≤ a ≤ 2
a
2
( )
G y =−e
−
1 2
y
2
( = w T x)
dJ y
dw
. Hyvarinen et al 2001 propose to use
© A.G.Billard 2004 – Last Update March 2011