MACHINE LEARNING TECHNIQUES - LASA

189
9.2.7 Statistical Independence
If y 1 and y 2 are two variables, generated by a random distribution p i (y i ) with zero-mean, then y 1
and y 2 are said to be mutually independent, if their joint density p(y 1 , y 2 ) is equal to the product of
their density functions:
pyy ( ) = py ( ) py ( )
(8.25)
1, 2 1 2
9.2.8 Uncorrelatedness
y 1 and y 2 are said to be uncorrelated, if their covariance is zero ( y y )
cov , 0
1 2
= . If the variables
are generated by a distribution with zero means, then uncorrelatedness is equal to product of
their separate expectations:
( ) ( ) ( )
cov( yy) = E yy − E y E y = 0
(8.26)
1, 2 1, 2 1 2
E( y y ) E( y ) E( y )
= (8.27)
1, 2 1 2
Independence is in general a much stronger requirement than uncorrelatedness. Indeed, if the y i
are independent, one has
E( f y f y ) = E( f y ) E( f y )
(8.28)
( ) ( ) ( ) ( )
1 2 1 2
for any functions f 1 and f 2 .
Independence and uncorrelatedness are equivalent when y 1 and y 2 have a joint Gaussian
distribution. The notion of independence depends, thus, on the definition of the probability
distribution of any given variables.
9.3 Information Theory
The mathematical formalism for measuring the information content, I, of an arbitrary
measurement I is due largely to a seminal 1948 paper by Claude E. Shannon. Within the context
of sending and receiving messages in a communication system, Shannon was interested in
finding a measure of the information content of a received message. Shannon’s measure of
information aimed at conveying the impression that an unlikely event contains more information
than a likely event. The information content of an outcome x is defined to be:
where P(x) is the probability of x.
( )
I x
1
= log2 =− log2
P( x)
(8.29)
P x
( )
© A.G.Billard 2004 – Last Update March 2011