MACHINE LEARNING TECHNIQUES - LASA

145
∂
∂
J 1
= fy −
1 2 λ 1yf
1 1
w1
f ( y 1y
)
1 2 λ 1
J 1
w y f x w f x y
v1
= −
∂
∂
w
2 2
= ( 1− ) −λ
( 1−
)
2
( 1 f ) ( y y )
1 x 2 λ 1
1 2 1 1 1 1 1 1 1
= − −
1 1 1
(6.51)
where in this last equation, all the vector multiplications are to be understood as being on an
element by element basis. Similarly with w 2 , v 2 , and λ 2 . This gives us a method for changing the
weights and the Lagrange multipliers on an online basis. This leads to the joint learning rules:
w
v
( )
f y λ y
Δ = η −
1 1 2 1 1
x w
1i 1i 1i
1
2
( y2 λ y1)( 1 f
1)
Δ = η − −
(6.52)
and similarly with the second set of weights.
6.7.3 ICA Revisited
An important aspect of anti-Hebbian learning is its ability to decorrelate inputs. As discussed in
Chapter 2.1.5, decorrelating the dataset if often too soft a constraint, and, it is often more
desirable to find independent components of the dataset, so as to reduce maximally its
dimensionaly, and, in the case of ANN, so as to maximize the information transmitted by the
network.
Jutten and Herault proposed a neural network architecture, based on anti-Hebbian learning, that
can perform Independent Component Analysis. The activation function is as follows:
Which is equivalent to:
i i ij j
j=
1
n
y x w y
= −∑ (6.53)
y= x− Wy
(6.54)
( ) 1
−
y= I+ W x
(6.55)
While this is similar to the Foldiak’s model, a crucial difference lies in the non-linearity of the
learning rule, defined as follows:
( ) ( ) for
Δ w =−α
f y g y i≠ j
(6.56)
ij i j
© A.G.Billard 2004 – Last Update March 2011