MACHINE LEARNING TECHNIQUES - LASA

138
There are two major advantages of using an ANN to process PCA as opposed to follow the
algorithm given in Section 2.1.5. It allows on-line as well as long-life learning Classical PCA
requires having at hand a sample of data point and runs in batch mode. ANNs can be run on-line.
They will discover the PCA components as data are fed onto them. This provides more flexibility
towards the data.
Let us now show that the classical Hebbian rule converges to the principal components of the
input vector:
Recall that
i i
Δ w
r =α yx
r and that y = w rr T x. Thus, we have:
i
i
Δ w r = α⋅y⋅ x r = α⋅ w r ⋅x r ⋅x
r
T
( )
(6.31)
T
Δ w r = α⋅ w r x r ⋅cos( θ)
⋅x
r
where θ is the angle between the weight vector and the input vector.
The weight vector moves proportionally to the distance across the weight vector and the input
vector: the bigger the distance, the bigger the move. Hebbian learning will, thus, minimize the
angle. If the data has zero mean, Hebbian learning will adjust the weight vector w r so as to
maximize the variance in the output y . In other words, when projected onto this direction given
by w r , the data will exhibit variance greater than in any other direction. This direction is, thus, the
largest principal component of the data and corresponds to the eigenvector of the correlation
matrix C with the largest corresponding eigenvalue.
If we decompose the current w r in terms of the eigenvectors of C,
expected weight update rule becomes:
r r
Δ w= α ⋅C⋅w
= α ⋅ ⋅
= α⋅
C ae
i i
i
∑
i
∑
r
aλe
i i i
r
r
w= ∑ae
i i
i
, then the
This will move the weight vector w r towards eigenvector e r i
by a factor proportional to aiλ . Over
i
many updates, the eigenvector with the largest λi
will drown out the contributions from the others.
© A.G.Billard 2004 – Last Update March 2011