In the Beginning was Information

Figure 31: Model of a discrete source for generating sequences of symbols.The source has a supply of N different symbols (e. g. an alphabet with 26 letters),of which a long sequence of n symbols is transmitted one after the other at a certaintime. The source could be a symbol generator which releases randomsequences of symbols according to a given probability distribution, or it could bean unknown text stored on magnetic tape which is transmitted sequentially (i. e.one symbol at a time).The first requirement (i) I(p) = I(p 1 x p 2 ) = I(p 1 ) + I(p 2 ) is met mathematicallywhen the logarithm of equation (1) is taken. The second requirement (ii) issatisfied when p 1 and p 2 are replaced by their reciprocals 1/p 1 and 1/p 2 :I (p 1 x p 2 ) = log(1/p 1 ) + log(1/p 2 ). (2)As yet the base b of the logarithms in equation (2) entails the question ofmeasure and is established by the third requirement (iii):I = log b (1/p) = log b (1/0.5) = log b 2 = 1 bit (3)It follows from log b 2 = 1 that the base b = 2 (so we may regard it as abinary logarithm, as notation we use log 2 = lb; giving lb x = (log x)/(log2); log x means the common logarithm that employ the base 10: log x =log 10 x). We can now deduce that the definition of the information contentI of one single symbol with probability p of appearing, is172I(p) = lb(1/p) = - lb p ≥ 0. (4)