In the Beginning was Information

of uncertainty which will be resolved when the next symbol arrives. Whenthe next symbol is a “surprise”, it is accorded a greater information valuethan when it is expected with a definite “certainty”. The reader who ismathematically inclined, may be interested in the derivation of some ofShannon’s basic formulas; this may contribute to a better understanding ofhis line of reasoning.1. The information content of a sequence of symbols: Shannon wasonly interested in the probability of the appearance of the various symbols,as should now become clearer. He thus only concerned himself withthe statistical dimension of information, and reduces the information conceptto something without any meaning. If one assumes that the probabilityof the appearance of the various symbols is independent of one another(e. g. “q” is not necessarily followed by “u”) and that all N symbols havean equal probability of appearing, then we have: The probability of anychosen symbol x i arriving is given by p i = 1/N. Information content is thendefined by Shannon in such a way that three conditions have to be met:i) If there are k independent messages 21 (symbols or sequences of symbols),then the total information content is given by I tot = I 1 + I 2 +…+ I k .This summation condition regards information as quantifiable.ii) The information content ascribed to a message increases when the elementof surprise is greater. The surprise effect of the seldom used “z”(low probability) is greater than for “e” which appears more frequently(high probability). It follows that the information value of a symbol x iincreases when its probability p i decreases. This is expressed mathematicallyas an inverse proportion: I ~ 1/p i .iii)In the simplest symmetrical case where there are only two differentsymbols (e. g. “0” and “1”) which occur equally frequently (p 1 = 0.5and p 2 = 0.5), the information content I of such a symbol will be exactlyone bit.According to the laws of probability the probability of two independentevents (e. g. throwing two dice) is equal to the product of the single probabilities:p = p 1 x p 2 (1)21 Message: In Shannon’s theory a message is not necessarily meaningful, but it refers toa symbol (e. g. a letter) or a sequence of symbols (e. g. a word). In this sense the conceptof a “message” is even included in the DIN standards system, where it is encodedas 44 300: “Symbols and continuous functions employed for the purpose of transmission,which represent information according to known or supposed conventions.”171