MACHINE LEARNING TECHNIQUES - LASA

139
6.6.4 Oja’s one Neuron Model
In the 1980’s Oja proposed a model that extracts the largest principal components from the input
data. The model uses a single output neuron with the classical activation rule:
Oja’s variation of the Hebbian rule is as follows:
y = ∑ wx
(6.32)
i
i
2
( )
i i i
i
Δ w = α x y− y w
(6.33)
2
Note that this rule is defined by a multiplicative constraint of the form y γ ( w)
= and so will
converge to the principal eigenvector of the input covariance matrix. The weight decay term has
the simultaneous effect of making ∑ ( w ) 2
i
tend towards 1, i.e. the weights are normalized.
i
This rule will allow the network to find only the first eigenvector. In order to determine the other
principal components, one must let the neurons interact with one another.
6.6.5 Convergence of the Weights Decay rule
The Hebbian rule with decay solves the problem of large weights. It eliminates very small and
irregular noise. However, it does so at a price. The environment must continuously present all
stimuli that have associations. Without reinforcement, associations will decay away. Moreover, it
takes longer to eliminate the effect of noise.
As mentioned earlier on, an important and distinctive function of the Hebbian learning rule is its
stability. Work by Miller & MacKay made an important contribution in showing the convergence of
the Hebbian learning rule with weight decay.
Let us consider the continuous time dependent learning rule with weight decay.
© A.G.Billard 2004 – Last Update March 2011