MACHINE LEARNING TECHNIQUES - LASA

144
= 1 2 1
2
{( ) + (1 ) (1 )}
1 2 1 1 2
2λ
− + 2λ
− (6.49)
2
J E yy y y
where the constraints have been implemented using Lagrange multipliers, λ 1 and λ 2 . Using gradient
ascent with respect to w i and gradient descent with respect to λ i gives the learning rules:
j( −λ
)
2
( ( y ) )
j( −λ
)
2
( ( y ) )
Δw ~ x y y
1j
1 2 1 1
Δλ
~ − 1−
1 1
Δw ~ x y y
2j
2 1 2 2
Δλ
~ − 1−
2 2
(6.50)
where w 1j is the j th element of weight vector w 1 etc.
However, just as a neural implementation of Principal Component Analysis (PCA) may be very
interesting but not a generally useful method of finding Principal Components, so a neural
implementation of CCA may be only a curiosity. It becomes interesting only in the light of performing
nonlinear CCA, which is described next.
6.7.2.1 Non-linear Canonical Correlation Analysis
To determine whether a neural network as described in the previous section can extract nonlinear
correlations between two data sets, x 1 and x 2 , and further to test whether such correlations are
greater than the maximum linear correlations, one must introduce a nonlinearity term to the
network. This takes the usual form of a tanh() function, which we also find in classical approach to
non-linear ANN approximation (e.g. as in feedforward NN with backpropagation).
The outputs y 1 and y 2 of the network become:
y
y
∑
( )
= w tanh v x = w
1 1j 1j 1j
1 1
j
∑
( )
= w tanh v x = w
2 2j 2j 2j
2 2
j
To maximise the correlation between y 1 and y 2 , we again use the objective function
whose derivatives give us
1 2 1
2
= E{( yy) + (1 − y) + (1 − y)}
2 2
J λ λ
1 1 2 1 1 2 2
f
f
© A.G.Billard 2004 – Last Update March 2011