MACHINE LEARNING TECHNIQUES - LASA

128
variance of the output, e.g. by increasing the weights. Note that, by doing this, you increase the
unpredictability of the neuron’s output, which can have disastrous consequences in some
applications.
Noise on the Inputs
In most applications, you will have to face noise in the input, with different type of noise
depending on the input. In such a scenario, the output of a neuron would be described by the
following:
( )
∑ j j j
(6.7)
j
y = w x + ν
One can show that the mutual information between input and output in this case becomes:
2
1 σ
y
I( x, y)
= log
2 ⎛⎛ ⎞⎞
2
σ
v ⎜⎜∑
wj⎟⎟
⎝⎝ j ⎠⎠
2
(6.8)
In this case, it is not sufficient to just increase the weights, since, by doing so, one will also
increase the amount on the denominator. More sophisticated techniques must be used on a
neuron-by-neuron basis.
More than one output neuron
Imagine a two input-output scenario in which the two outputs attempt to jointly convey as much
information as possible on the two inputs. In this case, each output’s neuron’s activation is given
by:
( )
∑ (6.9)
y = w x + ν
i ij j i
j
Similarly to the 1-output case, we can assume that the noise terms are uncorrelated and
Gaussian and we can write:
2
( ν) = ( ν1, ν2) = ( ν1) + ( ν2) = 1+
2log( 2πσ
)
h h h h
ν
Since the output neurons are both dependent on the same two inputs, they are correlated. One
can calculate the correlation matrix R as:
T
⎛⎛y1 ⎞⎞ ⎛⎛r11 r12
⎞⎞
R= E( yy ) = ⎜⎜ ⎟⎟( y1 y2)
= ⎜⎜ ⎟⎟
y r r
⎝⎝ 2⎠⎠ ⎝⎝ 21 22⎠⎠
(6.10)
One can show that the mutual information is equal to
© A.G.Billard 2004 – Last Update March 2011