MACHINE LEARNING TECHNIQUES - LASA

132
Figure 6-7: A difficult classification problem using a multi-layered feed-forward NN with increasing numbers
of neurons in the hidden layer. 3 neurons are sufficient to solve this problem.
[DEMOS\CLASSIFICATION\MLP-NEURONS.ML]
6.4.3 The Backpropagation Algorithm
1. Initialize the weights to small random numbers
p
2. Present an input patter X to the network
3. Compute
⎛⎛ ⎞⎞
p
1
ai = f ⎜⎜∑
wijxj
⎟⎟
⎝⎝ j ⎠⎠
and
⎛⎛ ⎞⎞
p
2
y = f ⎜⎜ wijaj
⎟⎟
⎝⎝ j ⎠⎠
∑ the outputs of the
hidden and output units respectively, where f is the activation function
1 2
(usually the sigmoid), and w ,
ij
w are the weights from 1 st layer, input to
ij
hidden units, and 2 nd layer, from hidden units to output units, respectively.
4. Compute the error, according to (6.12)
5. Compute the gradient of the error along each weight direction
second layer.
6. Compute the gradient of the error
where
s
1 p
i ij j
j
error to compute
∂E
∂E
∂s
∂w ∂s ∂w
p p
i
=
1 1
ij i ij
∂E
∂w
p
2
ij
for the
along the hidden units,
= ∑ w x . In order to do this, you need to backpropagate the
∂E
∂s
7. Update all weights according to (6.16)
i
p
.
8. Repeat all steps from 2 for all patterns and until the network has converged
(reached a minimal error).
Exercise: Show that a 1-layer feed-forward NN can be used to compute the XOR problem.
© A.G.Billard 2004 – Last Update March 2011