Biljeske 05 - Tehnički fakultet u Rijeci

TEORIJA INFORMACIJE
Željko Jeričević, dr. sc.
Zavod za računarstvo, Tehnički fakultet &
Zavod za biologiju i medicinsku genetiku, Medicinski fakultet
51000 Rijeka, Croatia
Phone: (+385) 51-651 594
E-mail: zeljko.jericevic@riteh.hr
http://www.riteh.uniri.hr/~zeljkoj/Zeljko_Jericevic.html

Information theory
Iz dosadašnjeg gradiva znamo da se informacija prije slanja kroz
kanal treba prirediti. To se postiže pretvorbom informacije u
formu koja ima entropiju blisku maksimalnoj čime se efikasnost
prenosa približava maksimalnoj. Ovo se može postići
kompresijom bez gubitaka informacije (lossless compression),
napr. Huffmanovim kodiranjem.
Druga pretvorba odnosi se na sigurnost prenosa pri čemu se
informacija prevodi u formu gdje je za određeni tip pogrešaka
moguća automatska korekcija (napr. Hamming-ovim
kodiranjem).
10 February 2012 zeljko.jericevic@riteh.hr 2

Sažimanje (compression)
10 February 2012 zeljko.jericevic@riteh.hr 3

Samuel F.B. Morse (1791-1872)
Slovo e je najčešće upotrebljavano
10 February 2012

Morse-ov kod
Točka je 1 bit, crta je 3 bita =>
razmak unutar istog slova je 1
bit, razmak između slova je 3
bita, razmak između riječi je 7
bitova.
Najkraći znak (slovo E) 1 bit,
najduži znak (broj 0) 19 bitova,
razmak među riječima 7 bitova.
Kako to komparira s ASCII?
Svi znakovi 8 bitova, razmak
među riječima 8 bitova. Za
točnu komparaciju potrebno je
znati učestalost znakova.

Zadatak
Izaberite engleski tekst po volji s projekta Gutemberg u
ASCII formatu (min 100KByte).
Odredite frekvenciju pojavljivanja svih ASCII znakova u
tekstu.
Izračunajte koliko bitova vam treba za Morseovu
reprezentacije teksta i usporedite s bitovima potrebnim
za ASCII reprezentaciju.

David A. Huffman (1925-1999)
David Huffman is best known for his legendary
Huffman code, a compression scheme for lossless
variable length encoding. It was the result of a term
paper he wrote while a graduate student at the
Massachusetts Institute of Technology (MIT), where
he earned a D.Sc. degree on a thesis named The
Synthesis of Sequential Switching Circuits, advised by
Samuel H. Caldwell (1953).
"Huffman Codes" are used in nearly every application
that involves the compression and transmission of
digital data, such as fax machines, modems, computer
networks, and high-definition television (HDTV), to
name a few.
From Wikipedia
10 February 2012 7

Shannon-Fano kodiranje
Top-down kodiranje (preteča Huffmanovog kodiranja)
“In Shannon–Fano coding, the symbols are arranged in
order from most probable to least probable, and then
divided into two sets whose total probabilities are as
close as possible to being equal. All symbols then have
the first digits of their codes assigned; symbols in the
first set receive "0" and symbols in the second set
receive "1". As long as any sets with more than one
member remain, the same process is repeated on those
sets, to determine successive digits of their codes.
When a set has been reduced to one symbol, of course,
this means the symbol's code is complete and will not
form the prefix of any other symbol's code.” from Wikipedia
8

Shannon-Fano kodiranje
9

Shannon-Fano kodiranje
Top-down kodiranje (preteča Huffmanovog kodiranja)
ABACADACABEDAADBECAEBADCAECAEBADBACABAD
Simbol A B C D E
Ucestalost 15 7 6 6 5
Vjerojatnost 0.38461538 0.17948718 0.15384615 0.15384615 0.12820513
10 February 2012 zeljko.jericevic@riteh.hr 10

Shannon-Fano kodiranje
Top-down kodiranje
Simbol A B C D E
Ucestalost 15 7 6 6 5
Vjerojatnost 0.39 0.18 0.15 0.15 0.13
Simbol A B C D E
Kod 00 01 10 110 111
2Bit
⋅ 15+ 7+ 6 + 3Bit
⋅ 6+
5
( ) ( )
39
≈
2.28 Bits / symbol
11

Huffman-ovo kodiranje
Bottom-up:
“D & E have the lowest frequencies and so
are allocated 0 and 1 respectively and
grouped together with a combined
probability of 0.28205128. The lowest pair
now are B and C so they're allocated 0 and
1 and grouped together with a combined
probability of 0.33333333. This leaves BC
and DE now with the lowest probabilities
so 0 and 1 are prepended to their codes
and they are combined. This then leaves
just A and BCDE, which have 0 and 1
prepended respectively and are then
combined. This leaves us with a single node
and our algorithm is complete.” From
Wikipedia
12

Huffman-ovo kodiranje
Bottom-up
Simbol A B C D E
Ucestalost 15 7 6 6 5
Vjerojatnost 0.39 0.18 0.15 0.15 0.13
Simbol A B C D E
Kod 0 100 101 110 111
1Bit
⋅ 15+ 3Bit
7+ 6+ 6+
5
39
( )
≈
2.23 Bits / symbol
13

Znak Ucestalost Kod
razmak 7 111
a 4 010
e 4 000
f 3 1101
h 2 1010
i 2 1000
m 2 0111
n 2 0010
s 2 1011
t 2 0110
l 1 11001
o 1 00110
p 1 10011
r 1 11000
u 1 00111
x 1 10010
Huffman-ovo kodiranje
"this is an example of a
huffman tree"
14

Huffman-ovo kodiranje
15

Huffman-ovo kodiranje
16

Huffman-ovo kodiranje
17

Huffman-ovo kodiranje
18

Huffman-ovo kodiranje
19

Huffman-ovo kodiranje
From Malan
20

Huffman-ovo kodiranje
"this is an example of a
huffman tree"
21

Huffman-ovo kodiranje
"this is an example of a
huffman tree"
22

Huffman-ovo kodiranje
23

Huffman-ovo kodiranje
24

Huffman-ovo kodiranje
25

Huffman-ovo kodiranje
26

Huffman-ovo kodiranje
27

Huffman-ovo kodiranje
"this is an example of a
huffman tree"
Znak p Huffman
A 0.2 ?
B 0.1 ?
C 0.1 ?
D 0.15 ?
E 0.45 ?
28

Huffman-ovo kodiranje
"this is an example of a
huffman tree"
Znak p Huffman
A 0.2 01
B 0.1 0000
C 0.1 0001
D 0.15 001
E 0.45 1
29

Modificirano Huffman-ovo kodiranje
Modificirano Huffmanovo kodiranje se koristi u
fax mašinama za kodiranje crnog na bijeloj
podlozi (bitmape). Kombinira Huffman-ove
kodove varijabilne duljine s repetitivnim
kodiranjem.
Za kodiranje crnog na bijelom, 1 bit po pikslu
(bijeli bitovi imaju vrijednost 0, crni bitovi imaju
vrijednost 1). Repeticije crnih i bijelih pikslova se
izbroje i pošalju kao Huffmanovi kodovi
varijabilne duljine.
30

Modificirano Huffmanovo kodiranje:
CCITT (Huffman) Encoding
“CCITT (International Telegraph and Telephone
Consultative Committee) is a standards organization that
has developed a series of communications protocols for the
facsimile transmission of black-and-white images over
telephone lines and data networks. These protocols are
known officially as the CCITT T.4 and T.6 standards but
are more commonly referred to as CCITT Group 3 and
Group 4 compression, respectively.”
31

CCITT Encodings
“Group 3 and Group 4 encodings are compression
algorithms that are specifically designed for
encoding 1-bit image data. Many document and
FAX file formats support Group 3 compression,
and several, including TIFF, also support Group 4.
”
32

CCITT Encodings
“Group 3 encoding was designed specifically for bilevel,
black-and-white image data
telecommunications. All modern FAX machines
and FAX modems support Group 3 facsimile
transmissions. Group 3 encoding and decoding is
fast, maintains a good compression ratio for a wide
variety of document data, and contains information
that aids a Group 3 decoder in detecting and
correcting errors without special hardware.”
33

CCITT Encodings
“Group 4 is a more efficient form of bi-level
compression that has almost entirely replaced the
use of Group 3 in many conventional document
image storage systems. (An exception is facsimile
document storage systems where original Group 3
images are required to be stored in an unaltered
state.)”
34

CCITT Encodings
“Group 4 encoded data is approximately half the
size of 1-dimensional Group 3-encoded data.
Although Group 4 is fairly difficult to implement
efficiently, it encodes at least as fast as Group 3
and in some implementations decodes even faster.
Also, Group 4 was designed for use on data
networks, so it does not contain the
synchronization codes used for error detection
that Group 3 does, making it a poor choice for an
35
image transfer protocol. ”

CCITT Encodings
“Group 3 normally achieves a compression ratio
of 5:1 to 8:1 on a standard 200-dpi (204x196 dpi),
A4-sized document. Group 4 results are roughly
twice as efficient as Group 3, achieving
compression ratios upwards of 15:1 with the same
document. Claims that the CCITT algorithms are
capable of far better compression on standard
business documents are exaggerated--largely by
hardware vendors.”
36

CCITT Encodings
“Because the CCITT algorithms have been optimized for
type and handwritten documents, it stands to reason that
images radically different in composition will not
compress very well. This is all too true. Bi-level bitmaps
that contain a high frequency of short runs, as typically
found in digitally half-toned continuous-tone images, do
not compress as well using the CCITT algorithms. Such
images will usually result in a compression ratio of 3:1 or
even lower, and many will actually compress to a size
larger than the original.”
37

CCITT Encodings
“The CCITT actually defines three algorithms for
the encoding of bi-level image data:
• Group 3 One-Dimensional (G31D)
• Group 3 Two-Dimensional (G32D)
• Group 4 Two-Dimensional (G42D)
”
38

Modificirano Huffman-ovo kodiranje
Svaka linije je kodirana kao izmjenjujući
nizovi crnih i bijelih bitova. Nizove duljine
63 ili manje kodirani su takozvanim
završnim kodom (termination code). Nizovi
duljine 64 ili više imaju početni (makeup
code) ispred završnog koda.
39

Modificirano Huffman-ovo kodiranje
Kodovi su određeni unaprijed, prema
reprezentativnoj statistici za printane
dokumente (85% bijelo, 15% crno; kraće
crne sekvence su vjerojatnije od bijelih, duže
bijele sekvence su vjerojatnije od crnih;
svaki redak dokumenta počinje s bijelom
sekvencom).
40

Modificirano Huffman-ovo kodiranje
Svaka linije je kodirana kao izmjenjujući
nizovi crnih i bijelih bitova. Nizove duljine
63 ili manje kodirani su takozvanim
završnim kodom (termination code). Nizovi
duljine 64 ili više imaju početni (makeup
code) ispred završnog koda.
41

Modificirano Huffman-ovo kodiranje
završni kodovi
Run Length White bits Black bits
0 00110101 0000110111
1 000111 010
2 0111 11
3 1000 10
4 1011 011
5 1100 0011
6 1110 0010
7 1111 00011
8 10011 000101

Modificirano Huffman-ovo kodiranje
završni kodovi
Run Length White bits Black bits
11 01000 0000101
.
17 101011 0000011000
.
28 0011000 000011001100
.
63 00110100 000001100111
43

Modificirano Huffman-ovo kodiranje
(bijeli i crni) početni kodovi
64 11011 000000111
128 10010 00011001000
192 010111 000011001001
256 0110111 000001011011
320 00110110 000000110011
.
1600 010011010 0000001011011
1664 011000 0000001100100
1728 010011011 0000001100101
1792 00000001000 00000001000
… 2560

Modificirano Huffman-ovo kodiranje
specijalni kodovi
“Several special code words are also defined in a
Group 3-encoded data stream. These codes are
used to provide synchronization in the event that a
phone transmission experiences a burst of noise.
By recognizing this special code, a CCITT decoder
may identify transmission errors and attempt to
apply a recovery algorithm that approximates the
lost data.”
45

Modificirano Huffman-ovo kodiranje
specijalni kodovi
“The EOL code is a 12-bit code word that begins each line
in a Group 3 transmission. This unique code word is used
to detect the start/end of a scan line during the image
transmission. If a burst of noise temporarily corrupts the
signal, a Group 3 decoder throws away the unrecognized
data it receives until it encounteres an EOL code. The
decoder would then start receiving the transmission as
normal again, assuming that the data following the EOL
is the beginning of the next scan line. The decoder might
also replace the bad line with a predefined set of data,
46
such as a white scan line.”

A decoder also uses EOL codes for several purposes. It
uses them to keep track of the width of a decoded scan
line. (An incorrect scan-line width may be an error, or it
may be an indication to pad with white pixels to the
EOL.) In addition, it uses EOL codes to keep track of the
number of scan lines in an image, in order to detect a
short image. If it finds one, it pads the remaining length
with scan lines of all white pixels.
EOL is 000000000001
RTC (Return To Control) is 6 consecutive EOL codes and
signifies end of message transmition.
Modificirano Huffman-ovo kodiranje
specijalni kodovi

Modificirano Huffman-ovo kodiranje
primjeri
48

Modificirano Huffman-ovo kodiranje
primjeri
49

Modificirano Huffman-ovo kodiranje
primjeri
a) Potrebno je samo poslati završni kod za 20 crnih bitova:
00001101000
50

Modificirano Huffman-ovo kodiranje
primjeri
b) Šaljemo početni kod za 64 bijelih bitova (11011) i
završni kod za 36 bijelih bitova: 00010011
51

Modificirano Huffman-ovo kodiranje
primjeri
c) Šaljemo 4 početna koda: 3 za 2560 crnih bitova
(000000011111) i jedan za 1088 crnih bitova
(0000001110101) i završni kod za 32 crnih bitova:
000001101010
52

Modificirano Huffman-ovo kodiranje
primjeri
Počinjemo s crnim bitom (neuobičajeno), umećemo
sekvencu bijelih bitova dužine 0, zatim kod za 1 crni bit,
sljedi kod za 4 bijela bita, zatim kod za 2 crna bita, kod za
1 bijeli bit, kod za 1 crni bit, početni kod za 1216 bijelih
bitova i konačni kod za 50 bijelih bitova, EOL
0 bijelo 00110101
1 crno 010
4 bijelo 1011
2 crno 11
1 bijelo 0111
1 crno 010
1266 Bijelo 011011000 + 01010011
EOL 000000000001
53

CCITT Encodings
“The CCITT actually defines three algorithms for
the encoding of bi-level image data:
• Group 3 One-Dimensional (G31D)
• Group 3 Two-Dimensional (G32D)
• Group 4 Two-Dimensional (G42D)
”
54

G32D CCITT Encoding
“With Group 3 Two-Dimensional (G32D)
encoding, the way a scan line is encoded may
depend on the immediately preceding scan-line
data. Many images have a high degree of vertical
coherence (redundancy). By describing the
differences between two scan lines, rather than
describing the scan line contents, 2D encoding
achieves better compression.”
55

G32D CCITT Encoding
“The first pixel of each run length is called a changing
element. Each changing element marks a color transition
within a scan line (the point where a run of one color ends
and a run of the next color begins).
The position of each changing element in a scan line is
described as being a certain number of pixels from a
changing element in the current, coding line (horizontal
coding is performed) or in the preceding, reference line
(vertical coding is performed). The output codes used to
describe the actual positional information are called
Relative Element Address Designate (READ) codes.” 56

G32D CCITT Encoding
“Shorter code words are used to describe the color
transitions that are less than four pixels away from each
other on the code line or the reference line. Longer code
words are used to describe color transitions lying a
greater distance from the current changing element.
2D encoding is more efficient than 1-dimensional because
the usual data that is compressed (typed or handwritten
documents) contains a high amount of 2D coherence.”
57

G32D CCITT Encoding
“Because a G32D-encoded scan line is dependent on the
correctness of the preceding scan line, an error, such as a
burst of line noise, can affect multiple, 2-dimensionally
encoded scan lines. If a transmission error corrupts a
segment of encoded scan line data, that line cannot be
decoded. But, worse still, all scan lines occurring after it
also decode improperly.”
58

G32D CCITT Encoding
“To minimize the damage created by noise, G32D uses a
variable called a K factor and 2-dimensionally encodes K-
1 lines following a 1-dimensionally encoded line. If
corruption of the data transmission occurs, only K-1 scan
lines of data will be lost. The decoder will be able to
resync the decoding at the next available EOL code.”
59

G32D CCITT Encoding
“The typical value for K is 2 or 4. G32D data that is
encoded with a K value of 4 appears as a single block of
data. Each block contains three lines of 2D scan-line data
followed by a scan line of 1-dimensionally encoded data.”
60

G32D CCITT Encoding
“The K variable is not normally used in decoding the
G32D data. Instead, the EOL code is modified to indicate
the algorithm used to encode the line following it. If a 1
bit is appended to the EOL code, the line following is 1-
dimensionally encoded; if a 0 bit is appended, the line
following the EOL code is 2-dimensionally encoded. All
other transmission code word markers (FILL and RTC)
follow the same rule as in G31D encoding. K is only
needed in decoding if regeneration of the previous 1-
dimensionally encoded scan line is necessary for error
61
recovery. ”

CCITT Encodings
“The CCITT actually defines three algorithms for
the encoding of bi-level image data:
• Group 3 One-Dimensional (G31D)
• Group 3 Two-Dimensional (G32D)
• Group 4 Two-Dimensional (G42D)
”
62

G42D CCITT Encoding
“Group 4 Two-Dimensional (G42D) encoding was
developed from the G32D algorithm as a better 2D
compression scheme--so much better, in fact, that Group
4 has almost completely replaced G32D in commercial
use.”
63

G42D CCITT Encoding
“Group 4 encoding is identical to G32D encoding except
for a few modifications. Group 4 is basically the G32D
algorithm with no EOL codes and a K variable set to
infinity. Group 4 was designed specifically to encode data
residing on disk drives and data networks. The built-in
transmission error detection/correction found in Group 3
is therefore not needed by Group 4 data.”
64

G42D CCITT Encoding
“The first reference line in Group 4 encoding is an
imaginary scan line containing all white pixels. In G32D
encoding, the first reference line is the first scan line of
the image. In Group 4 encoding, the RTC code word is
replaced by an end of facsimile block (EOFB) code, which
consists of two consecutive Group 3 EOL code words.
Like the Group 3 RTC, the EOFB is also part of the
transmission protocol and not actually part of the image
data. Also, Group 4-encoded image data may be padded
out with fill bits after the EOFB to end on a byte
boundary.”
65

G42D CCITT Encoding
“Group 4 encoding will usually result in an image
compressed twice as small as if it were done with G31D
encoding. The main tradeoff is that Group 4 encoding is
more complex and requires more time to perform. When
implemented in hardware, however, the difference in
execution speed between the Group 3 and Group 4
algorithms is not significant, which usually makes Group
4 a better choice in most imaging system
implementations.”
66

CCITT documents
• "Standardization of Group 3 Facsimile Apparatus for Document
Transmission," Recommendation T.4, Volume VII, Fascicle VII.3,
Terminal Equipment and Protocols for Telematic Services, The
International Telegraph and Telephone Consultative Committee
(CCITT), Geneva, Switzerland, 1985, pp. 16-31.
• "Facsimile Coding Schemes and Coding Control Functions for
Group 4 Facsimile Apparatus," Recommendation T.6, Volume
VII, Fascicle VII.3, Terminal Equipment and Protocols for
Telematic Services, The International Telegraph and Telephone
Consultative Committee (CCITT), Geneva, Switzerland, 1985, pp.
40-48.
10 February 2012 zeljko.jericevic@riteh.hr 67

Hvala na pažnji
Željko Jeričević, dr. sc.
Zavod za računarstvo, Tehnički fakultet &
Zavod za biologiju i medicinsku genetiku, Medicinski fakultet
51000 Rijeka, Croatia
Phone: (+385) 51-651 594
E-mail: zeljko.jericevic@riteh.hr
http://www.riteh.uniri.hr/~zeljkoj/Zeljko_Jericevic.html
10 February 2012 zeljko.jericevic@riteh.hr 68