MACHINE LEARNING TECHNIQUES - LASA

140
where g is an arbitrary function dependent on the weight w.
One can consider two cases:
1) Multiplicative constraints
d wt ( ) = Cwt ( ) − g ( wt ( ))
(6.34)
dt
( ( )) γ ( )
( ) ( )
g w t = w t ⋅ w t
(6.35)
where the decay term is multiplied by the weight. In this case, the decay term can
be viewed as a feedback term, which limits the rate of growth of each weight.
The bigger the weight, the bigger the decay.
2) Substractive constraint
d wt ( ) = Cwt ( ) − γ ( wt ( )) wt ( )
(6.36)
dt
where the weight decay is proportional to the weight and multiplies the weight.
6.7 Anti-Hebbian learning
If inputs to a neural network are correlated, then each contains information about the other. In
I x; y > 0.
other words, there is redundancy in the information conveyed by the input and ( )
Anti-Hebbian learning is designed to decorrelate inputs. The ultimate goal is to maximize the
information that can be processed by the network. The less redundancy, the more information
and the less number of output nodes required for transferring this information.
Anti-Hebbian learning is also known as lateral inhibition, as the anti-learning occurs within
members of the same layer. The basic model is defined by:
Δ w =− α < y ⋅ y > (6.37)
ij i j
where the angle brackets indicate the ensemble average of values taken over all training
patterns. Note that this is a tremendous limitation of the system, as it forces global and off-line
learning.
© A.G.Billard 2004 – Last Update March 2011