In the Beginning was Information

a) the factor n, which indicates that the information content is directlyproportional to the number of symbols used. This is totally inadequate fordescribing real information. If, for example, somebody uses a spate ofwords without really saying anything, then Shannon would rate the informationcontent as very large, because of the great number of lettersemployed. On the other hand, if someone who is an expert, expresses theactual meanings concisely, his “message” is accorded a very small informationcontent.b) the variable H, expressed in equation (6) as a summation over the availableset of elementary symbols. H refers to the different frequency distributionsof the letters and thus describes a general characteristic of the languagebeing used. If two languages A and B use the same alphabet (e. g. theLatin alphabet), then H will be larger for A when the letters are more evenlydistributed, i. e. are closer to an equal distribution. When all symbols occurwith exactly the same frequency, then H = lb N will be a maximum.An equal distribution is an exceptional case: We consider the case whereall symbols can occur with equal probability, e. g. when noughts and onesappear with the same frequency as for random binary signals. The probabilitythat two given symbols (e. g. G, G) appear directly one after the other,is p 2 ; but the information content I is doubled because of the logarithmicrelationship. The information content of an arbitrary long sequence ofsymbols (n symbols) from an available supply (e. g. the alphabet) whenthe probability of all symbols is identical, i. e.p 1 = p 2 = ... = p N = p, is found from equation (5) to be:nI tot = ∑ lb(1/p i ) = n x lb(1/p) = - n x lb p. (7)i=1If all N symbols may occur with the same frequency, then the probabilityis p = 1/N. If this value is substituted in equation (7), we have the importantequationI tot = n x lb N = n x H. (8)2. The average information content of one single symbol in asequence: If the symbols of a long sequence occur with differing probabilities(e. g. the sequence of letters in an English text), then we are interestedin the average information content of each symbol in this sequence,or the average in the case of the language itself. In other words: What isthe average information content in this case with relation to the averageuncertainty of a single symbol?174