Channel Coding 1 - Communications Engineering

Channel Coding 1
Dr.-Ing. Dirk Wübben
Institute for Telecommunications and High-Frequency Techniques
Department of Communications Engineering
Room: N2300, Phone: 0421/218-62385
wuebben@ant.uni-bremen.de
Lecture
Tuesday, 08:30 – 10:00 in S1270
Exercise
Wednesday, 14:00 – 16:00 in N2420
Dates for exercises will be announced
during lectures.
www.ant.uni-bremen.de/courses/cc1/
Tutor
Matthias Woltering
Room: N2400
Phone 218-62392
woltering@ant.uni-bremen.de

Cehannl Ciodng 1
Dr.-Ing. Drik Webbün
Iuttitnse for Toliteuemmancoincs and Hgih-Fueecqrny Tequecinhs
Dmpeteanrt of Ccninoomumtias Eiinennegrg
Room: N2300, Phnoe: 0241/821-62538
wuebben@ant.uni-bremen.de
Lrtecue
Tduaesy, 08:30 – 10:00 in S1270
Exsricee
Wedasnedy, 14:00 – 16:00 in N2420
Daets for ercesxeis wlil be
acounennd duinrg lteuecrs.
Toutr
Mttahias Wletirong
Room: N2400
Phone 218-62392
woltering@ant.uni-bremen.de
www.buchstaben-vertauschen.de
www.ant.uni-bremen.de/courses/cc1/
2

Preliminaries
Master students:
Channel Coding I and Channel Coding II are elective courses
Written examination (alternatively oral exam with written part) at the end of each semester
Diplomstudenten
Kanalcodierung als (Wahl)pflichtfach, mündliche Prüfung mit schriftlichen Anteil
Wahlweise dreistündig (Prüfung nach 1 Semester)
oder sechsstündig (Prüfung nach 2 Semestern)
Documents
Script Kanalcodierung I/II by Kühn & Wübben (in German), these slides and tasks for exercises
are available in the internet http://www.ant.uni-bremen.de/courses/cc1/
Exercises
Take place on Wednesday, 14:00-16:00 in Room N2420
Dates will be arranged in the lesson and announced by mailing list
cc_at_ant.uni-bremen.de
Contain theoretical analysis and tasks to be solved in Matlab
3

Selected Literature
Books on Channel Coding
A. Neubauer, J. Freudenberger, V. Kühn: Coding Theory: Algorithms, Architectures and
Applications, Wiley
R.E. Blahut, Algebraic Codes for Data Transmission, Cambridge University Press, 2003
W.C. Huffman, V. Pless, Fundamentals of Error-Correcting Codes, Cambridge, 2003
S. Lin, D.J. Costello, Error Control Coding: Fundamentals and Applications, Prentice-Hall, 2004
J.C. Moreira, P.G. Farr: Essentials of Error-Control Coding, Wiley, 2006
R.H. Morelos-Zaragoza: The Art of Error correcting Coding, Wiley, 2 nd Edition, 2006
S.B. Wicker, Error Control Systems for Digital Communications and Storage, Prentice-Hall, 1995
B. Friedrichs, Kanalcodierung, Springer Verlag, 1996
M. Bossert, Kanalcodierung, B.G. Teubner Verlag, 1998
J. Huber, Trelliscodierung, Springer Verlag, 1992
Books on Information Theory
T. M. Cover, J. A. Thomas, Information Theory, Wiley, 1991
R.G. Gallager, Information Theory and Reliable Communication, Wiley, 1968
R. McEliece: The Theory of Information and Coding, Cambridge, 2004
R. Johannesson, Informationstheorie - Grundlagen der (Tele-)Kommunikation, Addison-Wesley,
1992
4

Selected Literature
General Books on Digital Communication
J. Proakis, Digital Communications, McGraw-Hill, 2001
J.B. Anderson, Digital Transmission Engineering, IEEE Press, 2005
B. Sklar, Digital Communications, Fundamentals and Applications, Prentice-Hall, 2003
contains 3 chapters about channel coding
V. Kühn, Wireless communications over MIMO Channels: Applications to CDMA and Multiple
Antenna Systems, John Wiley & Sons, 2006
contains parts of the script
K.D. Kammeyer, Nachrichtenübertragung, B.G. Teubner, 4 th Edition 2008
K.D. Kammeyer, V. Kühn, MATLAB in der Nachrichtentechnik, Schlembach, 2001
chapters about channel coding and exercises
Internet Resources
Lecture notes, Online books, technical publications, introductions to Matlab, …
http://www.ant.uni-bremen.de/courses/cc1/
• Further material (within university net): http://www.ant.uni-bremen.de/misc/ccscripts/
Google, Yahoo, …
5

Claude Elwood Shannon: A mathematical theory of
communication, Bell Systems Technical Journal, Vol. 27,
pp. 379-423 and 623-656, 1948
“The fundamental problem of communication is
that of reproducing at one point either exactly or
approximately a message selected at another
point.”
To solve that task, he created a new branch of applied mathematics:
information theory and/or coding theory
Examples for source – transmission – sink combinations
Mobile telephone – wireless channel – base station
Modem – twisted pair telephone channel – internet provider
…
6

Outline Channel Coding I
1. Introduction
Declarations and definitions, general principle of channel coding
Structure of digital communication systems
2. Introduction to Information Theory
Probabilities, measure of information
SHANNON‘s channel capacity for different channels
3. Linear Block Codes
Properties of block codes and general decoding principles
Bounds on error rate performance
Representation of block codes with generator and parity check matrices
Cyclic block codes (CRC-Code, Reed-Solomon and BCH codes)
4. Convolutional Codes
Structure, algebraic and graphical presentation
Distance properties and error rate performance
Optimal decoding with Viterbi algorithm
7

Outline Channel Coding II
1. Concatenated Codes
Serial Concatenation & Parallel Concatenation (Turbo Codes)
Iterative Decoding with Soft-In/Soft-Out decoding algorithms
EXIT-Charts, Bitinterleaved Coded Modulation
Low Density Parity Check Codes (LDPC)
2. Trelliscoded Modulation (TCM)
Motivation by information theory
TCM of Ungerböck, pragmatic approach by Viterbi, Multilevel codes
Distance properties and error rate performance
Applications (data transmission via modems)
3. Adaptive Error Control
Automatic Repeat Request (ARQ)
Performance for perfect and disturbed feedback channel
Hybrid FEC/ARQ schemes
8

Chapter 1. Introduction
Basics about Channel Coding
General declarations and different areas of coding
Basic idea and applications of channel coding
Structure of digital communication systems
Discrete Channel
Statistical description
Base band and band pass transmission
AWGN and fading channel
Discrete Memoryless Channel (DMC)
Binary Symmetric Channel (BSC and BSEC)
Examples for simple Error Correction Codes
Single Parity Check (SPC), Repetition Code, Hamming Code
9

Important terms:
General Declarations
Message Amount of transmitted data or symbols by the source
Information Part of message, which is new for the sink
Redundancy Difference of message and information, which is
unknown to the sink
Message = Information + Redundancy
Irrelevance Information, which is not of importance to the sink
Equivocation Information, not stemming from source of interest
Message is also transmitted in a distinct amount of time
Messageflow Amount of message per time
Informationflow Amount of information per time
Transinformation Amount of error-free information per time transmitted
from the source to the sink
10

Three Main Areas of Coding
Source coding (entropy coding)
Compression of the information stream so that no significant information is lost, enabling a
perfect reconstruction of the information
Thus, by eliminating superfluous and uncontrolled redundancy the load on the transmission
system is reduced (e.g. JPEG, MPEG, ZIP)
Entropy defines the minimum amount of necessary information
Channel coding (error-control coding)
Encoder adds redundancy (additional bits) to information bits in order to detect or even
correct transmission errors at the receiver
Cryptography
The information is encrypted to make it unreadable to unauthorized persons or to avoid
falsification or deception during transmission
Decryption is only possible by knowing the encryption key
Claude E. Shannon (1948): A Mathematical Theory of Communication
http://www.ant.uni-bremen.de/misc/ccscripts/
11

Basic Principles of Channel Coding
Forward Error Correction (FEC)
Added redundancy is used to correct transmission errors at the receiver
Channel condition affects the quality of data transmission
errors after decoding occur if the error-correction capability of the code is passed
No feedback channel is required
varying reliability, constant bit throughput
Automatic Repeat Request (ARQ)
Small amount of redundancy is added to detect transmission errors
retransmission of data in case of a detected error
feedback channel is required
Channel condition affects the throughput
constant reliability, but varying throughput
Hybrid FEC/ARQ: Combination to use advantages of both schemes
Chapter 3 of
Channel Coding II
12

Basic Idea of Channel Coding
Sequence of information symbols u i is grouped into blocks of length k
uu 0 1uk1 uu k k1u2k1u2k u2k1u3k1
block block block
each block is
encoded separately
u channel
x
k encoder
n
Information vector u of length k: u = [u0 u1 …uk-1 ]
Elements ui stem from finite alphabets of size q: ui {0, 1, 2, … , q-1} binary code for q = 2
Channel Coding
Create a code vector x of length n > k with elements xi {0, 1, 2, … , q-1} for the information
vector u by a bijective (i.e. u x) function: x = [x0 x1 …xn-1]
Code contains set of all code words x, i.e. x for all possible u
qk different information vectors u and qn different vectors x exist
due to the bijective mapping from u to x only qk < qn vectors x are used
•Codeis a subset
• Coder is the mapper
Challenge: Find a k-dimensional subset out of an n-dimensional space so that minimum
distance between elements within is maximized effects probability of detecting errors
Code rate Rc equals the ratio of length of the uncoded and the coded
sequence and describes the required expansion of the signal bandwidth
k
Rc
n
13

Visualizing Distance Properties with Code Cube
q=2, n=3 x = [x 0 x 1 x 2 ] code word, i.e. x no code word x
010 011
000
110 111
100
d min = 1
Code rate R c = 1
001
No error correction
No error detection
101
010 011
000
110 111
100
d min = 2
Code rate R c = 2/3
No error correction
001
Detection of single error
101
010 011
000
110 111
100
d min = 3
Code rate R c = 1/3
001
Correction of single error
Detection of 2 errors
101
14

Applications of Channel Coding
Importance of channel coding increased with digital communications
First use for deep space communications:
AWGN channel, no bandwidth restrictions, only few receivers (costs negligible)
Examples: Viking (Mars), Voyager (Jupiter, Saturn), Galileo (Jupiter), ...
Digital mass storage
Compact Disc (CD), Digital Versatile Disc (DVD), Digital Magnetic Tapes (DAT), hard disc, …
Digital wireless communications:
GSM, UMTS, LTE, LTE-A, WLAN (Hiperlan, IEEE 802.11), ...
Digital wired communications
Modem transmission (V.90, ...), ISDN, Digital Subscriber Line (DSL), …
Digital broadcasting
Digital Audio Broadcasting (DAB), Digital Video Broadcasting (DVB)
Depending on the system (transmission parameters) different channel coding schemes
are used
15

Structure of Digital Transmission System
analog
source
digital source
source
encoder
u
• Source transmits signal d(t) (e.g. analog speech signal)
• Source coding samples, quantizes and compresses analog signal
• Digital source: comprises analog source and source coding, delivers
digital data vector u = [u 0 u 1 … u k-1 ] of length k
16

Structure of Digital Transmission System
analog
source
source
encoder
digital source
u
channel
encoder
x
• Channel encoder adds redundancy to u so that errors in
x = [x 0 x 1 … x n-1 ] can be detected or even corrected at receiver
• Channel encoder may consist of several constituent codes
• Code rate: R c = k / n
17

Structure of Digital Transmission System
analog
source
source
encoder
digital source
• Modulator maps discrete vector x onto analog waveform
and moves it into the transmission band
• Physical channel represents transmission medium
– Multipath propagation intersymbol interference (ISI)
– Time varying fading, i.e. deep fades in complex
envelope
– Additive noise
• Demodulator: Moves signal back into baseband and
performs lowpass filtering, sampling, quantization
u
channel
encoder
x
y
modulator
demodulator
physical
channel
discrete channel
• Discrete channel: comprises analog part of
modulator, physical channel and analog part
of demodulator
• in input alphabet of discrete channel
• out output alphabet of discrete channel
18

Structure of Digital Transmission System
analog
source
source
encoder
digital source
Channel decoder:
• Estimation of u on basis of received vector y
• y need not to consist of hard quantized values {0,1}
• Since encoder may consist of several parts,
decoder may also consist of several modules
u
channel
encoder
u
channel
decoder
x
y
modulator
demodulator
physical
channel
discrete channel
19

Structure of Digital Transmission System
analog
source
source
encoder
digital source
sink
u
feedback channel
source
decoder
channel
encoder
u
channel
decoder
x
y
modulator
demodulator
physical
channel
discrete channel
Citation of Jim Massey:
“The purpose of the modulation system is to create a good discrete channel from the
modulator input to the demodulator output, and the purpose of the coding system is to
transmit the information bits reliably through this discrete channel at the highest practicable
rate.”
20

Overview of Data Transmission System
Digital source
Source
Digital sink
Sink
Source
encoder
Source
decoder
Source transmits signals (e.g. speech)
Source encoder samples, quantizes and
compresses analog signal
Channel encoder adds redundancy to
enable error detection or correction @ Rx
Modulator maps discrete symbols onto
analog waveform and moves it into the
transmission frequency band
Channel
encoder
Digital
Transmission
Channel
decoder
Modulator
Analogue
Transmission
Demodulator
Discrete channel
Physical
Channel
Physical channel represents transmission
medium: multipath propagation, time varying
fading, additive noise, …
Demodulator: moves signal back into baseband
and performs lowpass filtering, sampling,
quantization
Channel decoder: Estimation of info sequence
out of code sequence error correction
Source decoder: Reconstruction of analog signal
21

Input and Output Alphabet of Discrete Channel
xi in discrete yi out channel
x = [x 0 x 1 …x n-1 ] y = [y 0 y 1 …y n-1 ]
Discrete channel comprises analog parts of modulator and demodulator as well
as physical transmission medium
Discrete input alphabet in = {X 0 , ..., X |in|-1 } x i X
Discrete output alphabet out = {Y 0 , ..., Y |out|-1 } y i Y µ
Common restrictions in this lecture
Binary input alphabet (BPSK): in ={-1,+1} (corresponds to bits {1, 0}); Pr{X }= 0.5
Output alphabet depends on quantization (q-bit soft decision)
• No quantization (q ): out =
• Hard decision (1-bit): out = in reliability information is lost
Cardinality ||:
number of elements
22

Stochastic Description of Discrete Channel
Properties of discrete channel are described by
yi Yxi X Y X
Pr Pr
YX
i.e. the probability that the symbol y i =Y is
received when the symbol x i = X was transmitted
(transition probability conditional probability)
Description by transition diagram
General relations (restriction to discrete output alphabet):
Probabilities: Pr{X}, Pr{Y} 0 £ Pr{X}, Pr{Y} £1
Wahrscheinlichkeit
Joint probability of event (X ,Y ): Pr{X ,Y }
Verbundwahrscheinlichkeit
Transition probabilities: Pr{Y | X }
Übergangswahrscheinlichkeit
bedingte Wahrscheinlichkeit
X 0
X 1
X |in |-1
Pr{Y 1 | X 0}
Pr{Y 0 | X 0}
Y 0
Y 1
Y |out |-1
A-priori probability
Pr{X }
XY Y X X Pr , Pr Pr
X Y Y Pr Pr
23

Stochastic Description of Discrete Channel
General probability relations
Pr a 1
i
i
aa b
Pr Pr , j
i j
j
i j
Pr a , b 1
A-posteriori probabilities:
“nach der Beobachtung”
XY Pr Pr 1
X Y
in out
Y X YX X Y
Pr Pr , Pr Pr ,
X Y
in out
Pr
X Y
X Y
in out
X Y
Pr , 1
X Y
Y Pr ,
For statistical independent elements (y gives no information about x)
Pr X , Pr Pr
Pr X, YPrXPrYPr
Y X Y
X Y Pr
X
Pr Y Pr Y
Pr
completeness
Marginal probability
Rand-Wahrscheinlichkeit
Prob. that X was transmitted
when Y is received
Information about x
“after observing y”
24

Bayes Rule:
Bayes Rule
Conditional probability of the event b given the occurrence of event a
Relation of a-posteriori probabilities Pr{X n |Y m } and transition probabilities Pr{Y m |X n }
and
but
Pr ab , Pr b
Prba Prab
Pr a Pr a
PrY X PrX
Y
Attention
Pr Y , Pr
Pr
X Y
X Y
1
Pr Y Pr Y
X X
in in
Y
out
Y X
Pr 1
X Y
out out
X Y
Y Pr ,
Pr 1
Pr
Y Y
Pr
Pr
Y X “for each receive symbol Y with
probability one a symbol X in was
transmitted”
“for each transmitted X in with
probability one a symbol of out is
received”
25

Example
Discrete channel with alphabets in = out = {0 ,1}
Signal values X0 = Y0 = 0 and X1 = Y1 = 1 with
probabilities Pr{X0 } = Pr{X1 } = 0.5
Transition probabilities Pr{Y0 | X0} = 0.9 and Pr{Y1 | X1} = 0.8
Due to completeness remaining transition probabilities
Pr{Y | X}
Y0 Y1 S
0.1
1
0.8
1
Joint probabilities: Pr{Y m , X n }=Pr{Y m |X n }·Pr{X n }
Pr{Y, X}
Y0 Y1 S
X 0
0.9
X 0
0.45
0.05
0.5
X 1
0.2
X 1
0.1
0.4
0.5
S
1.1
0.9
S
0.55
0.45
1.0
Pr{X |Y}
Y0 Y1 X 0
Pr{Y 1|X 0}
Pr{Y 0|X 1}
X 1
X 0
0.818
0.111
Pr{Y 0|X 0}
0.1
X 1
0.182
0.889
0.9
0.2
0.8
Pr{Y 1|X 1}
A-posteriori probabilities
Pr{X n|Y m}= Pr{Y m, X n}/Pr{Y m}
S
1
1
26
Y 0
Y 1

Continuous Output Alphabet
Continuous output alphabet, if no quantization takes place at the receiver
not practicable, because not realizable on digital systems
but interesting from information theory point of view
Discrete probabilities Pr{Y } become probability density function p y ()
Other relations are still valid
Examples:
Pr X p , X d
Pr
y
Y
out
y
Y
Yp d
y
out
p X d
1
with quantization boarders for Y m : ,
Y Y
27

xi
g () t
T
Baseband Transmission
Time-continuous, band-limited signal x(t)
x t x g tiT Average Power per period
Average Energy of each (real) transmit symbol
Noise with spectral density (of real noise) F NN = N 0 /2
Power
x() t
channel channel
i
i
T s
nt ()
gR() t
Signal to Noise Ratio (after matched filtering)
yi
2
2
X 2 BEs Es / Ts E X
N 22 B N B N 2T
2
N 0 0 0 s
iTs
2
X
0 /2 N
E E T
s s
S/ N
2
X
2
N
E
N
2B
,
XX NN
Es 2
X
2
N
Sampling
0
s
2
f 1 T 2B
A s
f
28

xi
g () t
T
x() t
√ 2 · e jω0t
Bandpass Transmission
Transmit real part of complex signal shifted to a carrier frequency f0 x t 2Re
j0t x t e 2 x't cos t x''t
sin
t
xBP (t)
With x(t) the complex envelope of the data signal
Re{} results in two spectra around -f0 and f0 y () t x () t h () t n
() t
Received signal
Transformation to baseband
Analytical signal achieved by Hilbert transformation (suppress f < 0)
y t y t jy t
doubles spectrum for f >0
Low pass signal
Re xBP() t
channel channel
nt ()
y () t
j
BP 0 0
y t 1 y 0 te
BP BP BP
BP BP BP BP
TP 2 BP
j t
BP
y () t
BP
1√ 2 · e −jω0t
gR() t
yi
29

-f0 B
1 2
2 X
1 2
2 N
Equivalent Baseband Representation
,
X X N N
BP BP BP BP
Equivalent received signal
y t
1
xBP 2
t hBP t nBP
j0t t
e x t h t n
t
2
X
2
N
In this semester, only real signals are investigated (BPSK)
Signal is only given in the real part
Only the real part of noise is of interest
f0 B
2BE 2 E
S/ N
2BN 2 N
s s
0 0
E E
N
f
0 2
r
s s
2
0 /2 N
E
S/ N
N
2
X
2
N
Es 2
0
s
2
B
,
XX NN
X
2
N
f
30

x i
0.8
0.6
0.4
0.2
Probability Density Functions for AWGN Channel
signal-to-noise-ratio E s /N 0 = 2 dB
p | 1
p | 1
y|
x
n i
y i
1
pn( ) e
2
py
y|
x
0
-4 -2 0
ddd
2 4
2
N
2
2
2
N
2
1 ( yX )
yx | ( | ) exp 2 2
2
2
N N
p y X
0.8
0.6
0.4
0.2
signal-to-noise-ratio E s /N 0 = 6 dB
p | 1
p | 1
y|
x
py
y|
x
0
-4 -2 0
ddd
2 4
31

Signal-to-Noise-Ratio S/N
S Es Ts Es
N N 2 T N 2
Error Probability of AWGN Channel
0 s 0
Error Probability for antipodal modulation
|
P Pr error x 1 p | E T d p E T d
e i y x s s n s s
0 0
0 0
i y| x s s n s s
Pr error x 1 p | E T d p E T d
2
E s Ts
1
exp
2
d
2
2
2
N 0
N
E s : symbol energy
N 0/2: noise density
Es Ts,
Es Ts
32

Error Probability of AWGN Channel
2
With N N T the error probability becomes
0 2
N s
Pe
2
1 Es T
s
exp
2 d 2
2
2
N 0 N
2
1
Es T
s
exp
d
N N 0 T
s 0
0 T
s
Using the substitution
Es Ts N Ts
with d d N T
0
or
Es Ts N Ts
with
0
1 2 1 E
s
Pe e d
erfc
2 N
E N
s
0 2
0
1
Pe d
2
d
d N T
with error function complement
2
2
erfc 1erf e d
0 2 s
with Q-function
0 s
2
2E
2
2
s
e Q
1 1
2
N Q E 0
e d
erfc
s N02
2
2 2
33

Error Function and Error Function Complement
Error Function
2
2
erf e d erfc()
1.5
Error Function Complement
2
2
erfc 1erf e d
Limits
1erf 1
0erfc 2
0
erf(), erfc()
2
1
0.5
0
-0.5
-1
-1.5
-2
erf()
-3 -2 -1 0 1 2 3
34

Probability Density Function for Frequency Nonselective Rayleigh
Fading Channel
Mobile communication channel is affected by fading (complex envelope of
receive signals varies in time) Magnitude |α| is Rayleigh distributed
0.1
0.08
0.06
0.04
0.02
x i
i
r i
0
0 2
i
4
n i
y i
0.1
0.08
0.06
0.04
0.02
p
0
-4 -2 0
ri
2 4
2
2
2 2
s s
exp for 0
0 else
0.1
0.08
0.06
0.04
0.02
0
-4 -2 0
yi
2 4
35

P b
10 0
10 -2
10 -4
Bit Error Rates for AWGN and Flat Rayleigh Channel
(BPSK modulation)
17 dB
AWGN
Rayleigh
10
0 10 20 30
-6
E s / N 0 in dB
AWGN channel:
0
2 s / 0
1
Pb erfc
2
Es / N
Q E N
Flat Rayleigh fading channel:
P
b
1 E / N
1
2 1 /
s 0
EsN0 Channel coding for fading
channels essential
(time diversity)
36

If line-of-sight connection exist for α, its real part is non-central Gaussian distributed
Rice factor K determines power ratio between line-of-sight path and scattered paths
K = 0 : Rayleigh fading channel
K →∞: AWGN channel
(no fading)
Coefficient i
has Rayleigh distributed magnitude with average power 1
2
Relation between total average power and variance: P(1 K)
Magnitude is Rician distributed
Rice Fading Channel
xi
K
1
K
1
1 K
'
2
2
K
K I
2 2 0 2
p| | ( )
K 1
i 1K 1K
i
exp 2 for 0
0
else
a
ni
yi
37

Discrete Memoryless Channel (DMC)
Memoryless: y i depends on x i but not on x i- for ≠0: Pr{y i | x i } = Pr{y i =Y |x i =X }
Transition probabilities of a discrete memoryless channel (DMC)
yx y0 y1 yn1 x0 x1 xn1 yi xi
Probability, that exactly m errors occur at distinct positions in a sequence of n is given by
Pr m bit of n incorrect
m
P 1 P
n m
specific positions
Probability, that in a sequence of length n exactly m errors occur (at any place)
n m
n m
Pr merror in a sequence of length n Pe 1Pe m
! giving the number of possibilities to choose m
with
elements out of n different elements, without
m m! nm! regarding the succession (combinations)
n1
Pr Pr , , , , , , Pr
n n
e e
i0
arbitrary positions
38

Binary Symmetric Channel (BSC)
Binary Symmetric Channel (BSC)
Binary input with hard decision at receiver results
in an equivalent binary channel
Symmetric transmission errors Pe are independent
of transmitted symbol
1 E s
PrY0 X1PrY1 X0 Pe
erfc
2
N
0
Probability, that sequence of length n is received correctly
xy x0 y0 x1 y1 xn1 yn1xi yi Pe
Pr Pr , , , Pr 1
Probability, for incorrect sequence (at least one error)
Pr 1Pr 1 1P
n
nP
x y x y for nP e 1
e e
BSC models BPSK transmission over AWGN and hard-decision
System equation y xe (with modulo-2 sum and ei ={0,1})
n1
i0
X 0
X 1
n
1-P e
1-P e
x i
x i
P e
P e
n i
e i
Y 0
Y 1
39
y i
y i

Binary Symmetric Erasure Channel (BSEC)
Binary Symmetric Erasure Channel (BSEC)
Instead of performing wrong hard decisions with high
probability, it is often favorable to erase unreliable bits
In order to describe “erased” symbols, a third output
symbol is required
For BPSK we use zero, i.e. out = {-1, 0, +1}
Transmission diagram
Pe denotes the probability of a wrong decision
Pq describes the probability of an erasure
X
1-Pe-Pq 0
Y0 X 1
P e
P e
P q
P q
1-P e-P q
Y 2
Y 1
1
Pe Pq y x
Pr xy Pq y?
Pe y x
Y 0
Y 2
Y 1
Binary Erasure
Channel (BEC): P e=0
X 0
X 1
1-P q
P q
1-P q
P q
Y 0
Y 2
Y 1
40

Example: Single Parity Check Code (SPC)
Code word x = [x 0 x 1 … x n-1 ] contains information word u = [u 0 u 1 … u k-1 ] and one
additional parity bit p (i.e., n = k+1)
0 k 1
k1
i0
i
p u u u
Each information word u is mapped onto one code word x (bijective mapping)
Example: k = 3, n = 4, binary digits {0,1}:
x x x x u p u u u p
x 1 3 [ ]
0 2 0 1 2
How many information words u of length k = 3 exist? 2
How many binary words of length n = 4 exist?
Code table
k = 23 = 8
2n = 24 2
= 16
k = 23 = 8
2n = 24 = 16
23 = 8 possible code words (out of 24 = 16 words)
u
000
001
010
011
x
000 0
001 1
010 1
011 0
even parity: under modulo-2 sum, i.e. the sum of all
elements of a valid code word x is always 0
u
100
101
110
111
x
100 1
101 0
110 0
111 1
Set of all code words
={0000, 0011, 0101, 0110, 1001, 1010, 1100, 1111}
e.g., y = [0001] is not a valid code word
transmission error occurred
Code enables error detection (all errors of unequal weight) but no error correction
41

Example: Repetition Code
Code word x = [x 0 x 1 … x n-1] contains n repetitions of the information word u = [u 0]
Set of all code words for n = 4 is given by={0000, 1111}
Transmission over BSC with Pe = 0.1 receive vector y = [y0 y1 … yn-1 ]
Maximum Likelihood Decoding Majority Decoder for unequal n:
0
z n/2
uˆ0 arg max Pru0 yarg max Prx
yuˆ
[ uˆ0]
u{0,1}
x
1
z n/2
Error rate performance for n = 3 and n = 5:
3 2 3 3
Perr, n3Pr{2 errors} Pr{3 errors} (1 Pe) Pe Pe
30.90.0110.001 2 3 0.028
5 5 5 P P P P P P
3 4 5 2 3 4 5
err, n5(1 e) e (1 e) e e 0.0086
with
n1
z y
i0
Lower error probability,
but larger transmission
bandwidth required
i
42

Example: Systematic (7,4)-Hamming Code
Codeword x = g(u) consists of the information word u and the parity word p
x
Parity bits
x x x x x x x [ u p]
u u u u p p p
0 1 2 3 4 5 6 0 1 2 3 0 1 2
p u u u
p u u u
p u u u
0 1 2 3
1 0 2 3
2 0 1 3
2 4 = 16 possible code words (out of 2 7 = 128)
u
0000
0001
0010
0011
x
0000 000
0001 111
0010 110
0011 001
u
0100
0101
0110
0111
x
0100 101
0101 010
0110 011
0111 100
even parity under modulo-2
sum, i.e., the sum within each
circle is 0
u
1000
1001
1010
1011
x
1000 011
1001 100
1010 101
1011 010
u
1100
1101
1110
1111
x
1100 110
1101 001
1110 000
1111 111
Example: Information word u = [1 0 0 0] generates code word
x = [1 0 0 0 0 1 1] all parity rules are fulfilled
p 1
u 0
p 2
u 3
u 2
1
u 1
1
1 0 0
0
p 0
0
43

Basic Approach for Decoding the (7,4)-Hamming Code
Maximum Likelihood Decoding: Find that information word u whose encoding
(code word) x = g(u) differs form the received vector y in the fewest number of
bits
Example 1: Error vector e = [0 1 0 0 0 0 0] flips the second bit
leading to the receive vector y = [1 1 0 0 0 1 1]
Parity is not even in two circles (syndrome s =[s0 s1 s2 ] = [1 0 1])
1
Task: Find smallest set of flipped bits that can cause this
0
s
violation of parity rules
1
Question: Is there a unique bit that lies inside all unhappy circles and outside the
happy circles? If so, the flipping of that bit would account for the observed syndrom
Yes, if bit y 1 is flipped all parity rules are again valid decoding was successful
1
1 0 1
s 2
0
44
s 0

Basic Approach for Decoding the (7,4)-Hamming Code
Example 2: y = [1 0 0 0 0 1 0]
Only one parity check is violated
only y7 lies inside this unhappy circle but outside the happy circles
Flip y 7 (i.e. p 2 )
Example 3: y = [1 0 0 1 0 1 1]
All checks are violated, only y 3 lies inside all three circles
y 3 (i.e. u 3 ) is the suspected bit
Example 4: y = [1 0 0 1 0 0 1]
Two errors occurred (bits y 3 and y 5 )
Syndrome s = [1 0 1] indicates single bit error y 1
Optimal decoding flips y 1 (i.e. u 1 )
leading to decoding result with 3 errors
0
1
1 1 0
0
0
1
1
0
0
1 0 0
0
1
1 1 0
0
1
1 1 1
0
0
0
0
45

Error Rate Performance for BSC with P e = 0.1
With increasing n the BER decreases, but also the code rate R c throughput
10 0
10 -2
BER P b
10 -4
10 -6
10 -8
R(9)
R(5)
B(127,22)
R(13)
B(255,47)
B(255,45)
R(17) B(127,15)
B(255,37)
R(21)
R(25)
R(29)
R(n): RPC
H(n,k): Hamming-Code
B(n,k): BCH-Code
R(3)
B(63,7)
B(255,29)
B(511,76)
B(511,67)
B(127,36)
more useful
codes
achievable
H(7,4)
H(15,11)H(31,26)
H(63,57) H(127,120)
not achievable
0 0.2 0.4 0.6 0.8 1
Rate R c = k/n
R(1)
Trade-off between BER and rate
What points (R c , P b ) are achievable?
Shannon: The maximum rate R c at
which communication is possible
with arbitrarily small P b is called
the capacity of the channel
Capacity of BSC with P e =0.1 equals
C = 1 + P e ·log 2 P e + (1-P e )·log 2 (1-P e )]
0.53 bit
With R c > 0.53 no reliable communication
is possible for P e = 0.1!
46