MACHINE LEARNING TECHNIQUES - LASA

127
6.3.2 Information Theory and the Neuron
A good start to understand the properties of ANN is to look at those from the point of view of
information theory. Linsker applied information theory to the analysis of the processing of a single
neuron. He started by defining the infomax principle, which states that: the learning rule should be
such that the joint information between the inputs and output of the neuron is maximal.
As mentioned in the introduction, a fundamental principle of learning systems is their robustness
to noise. In a single neuron model, one way to measure the neuron’s robustness to noise is to
determine the joint information between its inputs and output.
Let us consider a neuron with noise on the output:
y ⎛⎛ ⎞⎞
= ⎜⎜ w x + ν
⎝⎝ ⎠⎠
∑ j j⎟⎟
(6.3)
j
Let us assume that the output, y, and the noise, ν follow a Gaussian distribution with zero-mean
and variance σ and σ respectively. Let us further assume that the noise is uncorrelated with
2
y
any of the inputs, i.e.
2
ν
E( ν x i ) = 0 ∀ i
(6.4)
We can now compute the mutual information between the neuron’s input and output:
( , ) ( ) ( | )
where h( z ) is the entropy of z , see Section 9.3.1.
I x y = h y − h y x
(6.5)
In other words, I(x,y) measures the information contained in the output about the input and is
equal to the information in the output minus the uncertainty in the output, when knowing the value
of the input.
Since the neuron’s activation function is deterministic, the uncertainty on the output is solely due
h y|
x = h ν . Taking into account the Gaussian properties of
to the noise. Hence, we have: ( ) ( )
the output and the noise, we have;
( , ) = ( ) − ( )
I x y h y h ν
=
1 log
2
σ
σ
2
y
2
v
(6.6)
σ
σ
2
y
2
v
stands for the signal/noise ratio of the neuron. Since the experimental set-up fixes the
amount and variance of noise, one can modulate only the variance of the output. In order to
improve the signal/noise ratio, and, thus, the amount of information, one can, thus, increase the
© A.G.Billard 2004 – Last Update March 2011