MACHINE LEARNING TECHNIQUES - LASA

143
Foldiak’s second model allows all neurons to receive their own outputs with weight 1.
( 1 y y )
Δ w = α − (6.46)
ii i i
which can be written in matrix form as
T
( )
Δ W = α I− YY
(6.47)
where I is the identity matrix.
This network will converge when the outputs are decorrelated (due to the off-diagonal anti-
Hebbian learning) and when the expected variance of the outputs is equal to 1. i.e. this learning
rule forces each network output to take responsibility for the same amount of information since
the entropy of each output is the same.
This is generalizable to
( y y )
Δ w = αθ− (6.48)
ij ij i j
Where
θ
ij
= 0 for i ≠ j. The value of θ
ii
for all i , will determine the variance on that output
and so we can manage the information output of each neuron.
6.7.2 CCA Revisited
Adapted from Peiling Lai and Colin Fyfe, Kernel and Nonlinear Canonical Correlation Analysis, Computing and
Information Systems, 7 (2000) p. 43-49.
The Canonical Correlation Network
Figure 1 The CCA Network. By adjusting weights, w 1 and w 2 , we maximize correlation
between y 1 and y 2 .
Let us consider CCA in artificial neural network terms. The input data comprises two vectors x 1 and
x 2 . Activation is fed forward from each input to the corresponding output through the respective
weights, w 1 and w 2 (see Figure 1 and equations (1) and (2)) to give outputs y 1 and y 2 .
One can derive an objective function for the maximization of this correlation under the constraint
that the variance of y 1 and y 2 should be 1 as:
© A.G.Billard 2004 – Last Update March 2011