MACHINE LEARNING TECHNIQUES - LASA

136
6.6.2 Weight decay
A more interesting option is to prune weights that seem to have little influence on the network
operation. This is an operation that requires global knowledge of all the weights in the network,
and, thus, is less supported biologically. The idea is that no single weight should grow too large,
while keeping the total weight connections into a particular output neuron fairly constant. One of
the simplest rule for weight decay was developed by Grossberg in 1968 and was of the form:
dw
dt
ij
= α yx − w
(6.23)
i j ij
dw ij
It is clear that the weights will be stable when 0
dt
= at the points where wij = αE( xi yj
)
One can show that, at stability, we must have α Cw = w and, thus, that w must be an
eigenvector of the correlation matrix C.
We will next consider two more sophisticated learning equations, which use weight decay.
The instar rule where the decay term is gated by the input term
In the discrete case, we have:
dw
dt
ij
{ }
i ij j
x
j
:
= α y − w x
(6.24)
Δ w = α⋅x ⋅y −γ
⋅w ⋅ y
(6.25)
ij i j ij j
( )
α = γ ⇒ Δ w = α⋅y ⋅ x − w
(6.26)
ij i i ij
( )
y = 1 ⇒ Δ w = α ⋅ x −w
i ij i ij
( ) (1 α) ( 1) α ( )
⇒ w t = − ⋅w t− + ⋅x t
ij ij i
(6.27)
The weight moves in the direction of the input.
© A.G.Billard 2004 – Last Update March 2011