In the Beginning was Information

The corresponding value for English is H 1 = 4.04577 bits per letter. Weknow that the probability of a single letter is not independent of the adjacentletters. Q is usually followed by u, and, in German, n follows e muchmore frequently than does c or z. If we also consider the frequency ofpairs of letters (bigrams) and triplets (trigrams), etc., as given in Table 4,then the information content as defined by Shannon, decreases statisticallybecause of the relationships between letters, and we have:H 0 > H 1 > H 2 > H 3 > H 4 > ... > H ∞ . (13)With 26 letters the number of possible bigrams is 26 2 = 676, and therecould be 26 3 - 26 = 17,550 trigrams, since three similar letters are neverconsecutive. Taking all statistical conditions into consideration, Küpfmüller[K4] obtained the following value for the German language:H ∞ = 1.6 bits/letter. (14)For a given language the actual value of H 0 is far below the maximumvalue of the entropy. The difference between the maximum possible valueH max and the actual entropy H, is called the redundance R . The relativeredundance is calculated as followsr = (H max - H)/H max . (15)For written German, r is given by (4.755 – 1.6)/4.755 = 66%. Brillouinobtained the following entropy values for English [B5]:= 4.03 bits/letter,= 3.32 bits/letter,= 3.10 bits/letter,H ∞ = 2.14 bits/letter.H 1H 2H 3We find that the relative redundance for English, r = (4.755 - 2.14)/4.755= 55 % is less than for German. In Figure 32 the redundancy of a languageis indicated by the positions of the different points.Languages usually employ more words than are really required for fullcomprehensibility. In the case of interference certainty of reception isimproved because messages usually contain some redundancy (e. g. illegiblywritten words, loss of signals in the case of a telegraphic message, orwhen words are not pronounced properly).2. Syllables: Statistical analyses of the frequencies of German syllableshave resulted in the following value for the entropy when their frequencyof occurrence is taken into account [K4]:197