MACHINE LEARNING TECHNIQUES - LASA

141
If
y
i
and
y are highly correlated, then, the weights between them will grow to a large negative
j
value and each will tend to turn the other off. Indeed, we have:
( Δwij →0 ) ⇒ ( < yi , yj
> → 0)
The weight change stops when the two outputs are decorrelated. At this stage, the algorithm
converges. Note that there is no need for weight decay or renormalizing on anti-Hebbian weights,
as they are automatically self-limiting.
6.7.1 Foldiak’s models
Foldiak has suggested several models combining anti-Hebbian learning and weight decay. Here,
we will consider the first 2 models as examples of solely anti-Hebbian learning. The first model is
shown in Figure 6-12 and has anti-Hebbian connections between the output neurons.
Figure 6-12: Foldiak's 1st model
The equations, which define its dynamical behavior, are
with learning rule
y x w y
i i ij j
j=
1
n
= +∑ (6.38)
Δ w =−α
⋅y ⋅y for i≠ j
(6.39)
ij i j
In matrix terms, we have
And so,
y= x+ W⋅y
( ) −1
y= I −W ⋅x
(6.40)
Therefore, we can view the system as a transformation, T, from the input vector x to the output
y given by:
( ) 1
−
y= T⋅ x= I−W ⋅ x
(6.41)
Now, the matrix W must be symmetric. It has only non-zero non-diagonal terms, i.e. if we
consider only a two input, two output net as in the diagram.
W
⎛⎛0
w⎞⎞
= ⎜⎜ ⎟⎟
⎝⎝ w 0⎠⎠
© A.G.Billard 2004 – Last Update March 2011