MACHINE LEARNING TECHNIQUES - LASA

191
1 1 1 1
h x ∫ a b a b
0
a a
∫ a
b b
a
a+
b
( ) =− log − log = log + log = log ( ⋅ )
This definition is appealing, as, if a or b have a large value, the spread of the distribution will be
large.
One can show that the Gaussian distribution is the distribution with maximal entropy for a given
variance. In other words, this means that the gaussian distribution is the ``most random'' or the
least structured of all distributions. Entropy is small for distributions that are clearly concentrated
on certain values, i.e., when the variable is clearly clustered, or has a probability distribution
function that is very ``spiky''.
9.3.2 Joint and conditional entropy
If x and y are two random variables taking values between [0, I] ∈• and [0, J]
∈•
respectively, and pi (, j ) the joint probability that x takes value I and y takes value j, then the
entropy of the joint distribution is equal to:
=−∑ (8.32)
H( x, y) P( i, j)log P( i, j)
i,
j
We can then derive the equation for the conditional entropy.
∑
H( x| y) =− P( j) H( x| y = j)
j
∑
= − P( j) P( i| j)log P( i| j)
j
∑
= − Pi ( , j)log Pi ( | j)
i,
j
∑
i
Finally, conditional and joint entropy can be related as follows:
( ) ( )
Hxy (, ) = H x+ H yx |
(8.33)
Moreover, if the two variables are independent, then the entropy is additive:
H( x, y) = H( x) + H( y) iff P( x, y) = P( x) P( y)
9.3.3 Mutual Information
The mutual information between two random variables x and y is denoted by
© A.G.Billard 2004 – Last Update March 2011