MACHINE LEARNING TECHNIQUES - LASA

37
Denote by g the derivative of the above two functions, we have:
( ) = tanh ( )
g u au
1 1
2
( )
g u = ue
⎛⎛
2
u ⎞⎞
⎜⎜
⎜⎜
− ⎟⎟
2 ⎟⎟
⎝⎝ ⎠⎠
1≤a
≤2is some suitable constant, often taken as a 1 =1. These function are monotonic
where
1
and hence particularly well suited for performing gradient descent.
The basic form of the FastICA algorithm is as follows:
1. Choose an initial (e.g. random) weight vector W.
T
2. Compute the quantity ( )
T
{ } { ( )}
w + = E xg w x − E g w x w
3. Proceed to a normalization of the weight vector:
w =
+
w
+
w
r
−1 ⋅ ≠1
4. If the weights have not converged, i.e. wt ( ) wt ( )
r
, go back to step 2.
Note that it is not necessary that the vector converge to a single point, since w and -w define the
same direction. Recall also that it is here assumed that the data have been whitened.
2.3.6.3 FastICA for several units
The one-unit algorithm of the preceding subsection estimates just one of the independent
components, or one projection pursuit direction. To estimate several independent components,
we need to run the one-unit FastICA algorithm using several units (e.g. neurons) with weight
w w .
vectors
1 ,..., q
To prevent different vectors from converging to the same maxima we must decorrelate the
outputs T ,..., T
w1 w at each iteration. We present here three methods for achieving this.
q
A simple way of achieving decorrelation is a deflation scheme based on a Gram-Schmidt-like
decorrelation. This means that we estimate the independent components one by one. When we
have estimated p independent components, or p vectors w ,..., 1
w
p
, we run the one-unit fixedpoint
algorithm for w
p + 1, and after every iteration step subtract from w
p + 1
the ``projections''
+
= of the previously estimated p vectors, and then renormalize w
p + 1:
T
wp 1
wjwj, j 1,... p
1. Let
p
T
p+ 1
=
p+ 1−∑
p+
1 j j
j=
1
w w w w w
2. Let w = w / w w
T
p+ 1 p+ 1 p+ 1 p+
1
(2.32)
© A.G.Billard 2004 – Last Update March 2011