MACHINE LEARNING TECHNIQUES - LASA

190
Shannon’s information measure is binary coded. To get an intuitive feeling of this concept,
consider the case where the probability of x is uniform over the interval [1, 2]∈• . Then, the
information conveyed in the observed event x=1 is equal to:
1
I( x= 1) =− log2 = log22=
1
2
If, on the other hand, x is uniformly distributed over the interval [1,8]∈• , making each of the
occurrences of x 3 times less likely. Then, the information conveyed in observing x=1, is 3 times
more important:
( )
3
I x= 1 = log 8= log 2 = 3
2 2
9.3.1 Entropy
The notion of information is tightly linked to the physics notion of entropy. The entropy of the
information on x is a measure of the uncertainty or of the information content of a set of N
observations of x:
N
=−∑ (8.30)
( ) ( ) log ( )
H x P x P x
i=
1
If we take again our earlier example, the entropy of x, when x is uniform over the interval [1,2] is
equal to
( )
H x
2
1 1
=− ∑ log2
= 1
2 2
i=
1
. If now, we choose a non uniform distribution of probability for x, wher
H x
⎛⎛1 1 3 3⎞⎞
=− ⎜⎜ log + log ⎟⎟=
0.8
⎝⎝4 4 4 4⎠⎠
e P(x=1)=1/4 and P(x=2)=3/4, then ( ) 2 2
Thus, there is more uncertainty in a situation where every outcome is equally probable.
.
In the continuous case, one defines the differential entropy of a distribution f of a random variable
x as:
h x =−∫ f x log f x dx
(8.31)
( ) ( ) ( )
S
where S is the support set of x. For instance, consider the case where x is uniformly distribution
between [0,a], so that its density function is
Then, its differential entropy is:
f
( x)
⎧⎧1
if 0 ≤ x≤a
⎪⎪a
= ⎨⎨
⎪⎪ 1
if a < x≤ a+
b
⎪⎪⎩⎩ b
,
© A.G.Billard 2004 – Last Update March 2011