In the Beginning was Information

According to Shannon’s definition the information content of a singlemessage (whether it is one symbol, one syllable, or one word) is a measureof the uncertainty of its reception. Probabilities can only have valuesranging from 0 to 1 (0 ≤ p ≤1), and it thus follows from equation (4) thatI(p) ≥ 0, meaning that the numerical value of information content isalways positive. The information content of a number of messages (e. g.symbols) is then given by requirement (i) in terms of the sum of the valuesfor single messagesnI tot = lb(1/p 1 ) + lb(1/p 2 ) +...+ lb(1/p n ) = ∑ lb(1/p i ) (5)i=1As shown in [G7] equation (5) can be reduced to the following mathematicallyequivalent relationship:NI tot = n x ∑ p(x i ) x lb(1/(p(x i )) = n x H (6)i=1Note the difference between n and N used with the summation sign ∑. Inequation (5) the summation is taken over all n members of the receivedsequence of signs, but in (6) it is summed for the number of symbols N inthe set of available symbols.Explanation of the variables used in the formulas:n = the number of symbols in a given (long) sequence (e. g. the totalnumber of letters in a book)N = number of different symbols available(e.g: N = 2 for the binary symbols 0 and 1, and for the Morse codesymbols and –N = 26 for the Latin alphabet: A, B, C,..., ZN = 26 x 26 = 676 for bigrams using the Latin alphabet: AA,AB, AC,..., ZZN = 4 for the genetic code: A, C, G, Tx i ; i = 1 to N, sequence of the N different symbolsI tot = information content of an entire sequence of symbolsH = the average information content of one symbol (or of a bigram, ortrigram; see Table 4); the average value of the information content ofone single symbol taken over a long sequence or even over the entirelanguage (counted for many books from various types of literature).Shannon’s equations (6) and (8) used to find the total (statistical!) informationcontent of a sequence of symbols (e. g. a sentence, a chapter, or abook), consist of two essentially different parts:173