In the Beginning was Information

To compute the average information content per symbol I ave , we have todivide the number given by equation (6) by the number of symbols concerned:NI ave = I tot /n = ∑ p(x i ) x lb(1/p(x i )). (9)i=1When equation (9) is evaluated for the frequencies of the letters occurringin English, the values shown in Table 1, are obtained. The average informationcontent of one letter is I ave = 4.045 77. The corresponding value forGerman is I ave = 4.112 95.The average I ave (x) which can be computed from equation (9) thus is thearithmetic mean of the all the single values I(x). The average informationcontent of every symbol is given in Table 1 for two different symbol systems(the English and German alphabets); for the sake of simplicity i is usedinstead of I ave . The average information content for each symbol I ave (x) ≡ i isthe same as the expectation value 22 of the information content of one symbolin a long sequence. This quantity is also known as the entropy 23 H of thesource of the message or of the employed language (I ave ≡ i ≡ H). Equation(9) is a fundamental expression in Shannon’s theory. It can be interpretedin various ways:a) Information content of each symbol: H is the average information contentI ave (x) of a symbol x i in a long sequence of n symbols. H thus is acharacteristic of a language when n is large enough. Because of the differentletter frequencies in various languages, H has a specific value for everylanguage (e. g. H 1 = 4.045 77 for English and for German it is 4.112 95).22 Expectation value: The expectation value E is a concept which is defined for randomquantities in probability calculus. The sum ∑ p k x g(x k ) taken over all k singlevalues, is called the expectation value E of the probability distribution, where g(x) isa given discrete distribution with x k as abscissae and p k as ordinates (= the probabilityof appearance of the values x k ). This value is also known as the mean value or themathematical hope.23 Entropy: This concept was first introduced in thermodynamics by Rudolf Clausiusabout 1850. Later, in 1877, Ludwig Boltzmann (1844-1906) showed that entropy isproportional to the logarithm of the probability of a system being in a certain state.Because the formal derivation of the mathematical formulas for physical entropy issimilar to equation (9), Shannon (1948) also called this quantity entropy. Unfortunatelythe use of the same term for such fundamentally different phenomena hasresulted in many erroneous conclusions. When the second law of thermodynamics,which is also known as the entropy theorem, is flippantly applied to Shannon’s informationconcept, it only causes confusion. In thermodynamics entropy depends onthe temperature, which can not at all be said of informational entropy.175