6.451 Principles of Digital Communication II Wednesday, May 4 ...

6.451 Principles of Digital Communication II Wednesday, May 4, 2005MIT, Spring 2005 Handout #22Problem Set 9 SolutionsProblem 8.3 (revised) (BCJR (sum-product) decoding of SPC codes)As shown in Problem 6.4 or Figure 1 of Chapter 10, any (n, n − 1, 2) single-parity-checkcode has a two-state trellis diagram. Consider the (8, 7, 2) code, and let the receivedsequence from a discrete-time AWGN channel with input alphabet {±1} and noise variance2σ =1 be given by r =(0.1, −1.0, −0.7, 0.8, 1.1, 0.3, −0.9, 0.5). Perform BCJR (sumproduct)decoding of this sequence to determine the APP vector for each symbol Y k . [Hint:the special algorithm given in Section 13.5.2 (see Problem 9.5) to implement the sumproductupdate rule for zero-sum nodes may be helpful here.]First, let us draw the normal graph of a minimal trellis realization of the (8, 7, 2) code.This graph is shown abstractly below:Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7B0Σ1 BΣ 2 Σ 3 Σ 4 Σ 5 Σ 6 Σ 71 B 2 B 3 B 4 B 5 B 6 B 7(2, 1) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (2, 1)Moreover, it is easy to see that the (3, 2) branch constraint codes are all (3, 2, 2) zero-sumcodes, and the (2, 1) codes are simple repetition codes that need not actually be shown.Therefore the trellis realization may be drawn simply as follows:Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7+ + + + + +Note that this trellis realization of the (8, 7, 2) code is another cycle-free realization thatuses 6 (3, 2, 2) zero-sum constraints, as in the reduced HT realization of Problem 8.1;however, the diameter of this realization is 5 (which is as large as it could possibly be).For a binary-input Gaussian-noise channel with inputs {±1} and Gaussian conditionalprobability density p(r | y) =(2πσ) −1/2 exp −(r − y) 2 /2σ 2 , the a posteriori probability(APP) of y ∈ {±1} given a received value r is, by Bayes’ rule,p(r | y)p(y | r) = p(r | 1) + p(r | −1)e ry/σ2= .er/σ 2 + e−r/σ 2Here σ 2 = 1 (so SNR = 1 (0 dB); i.e., the channel is very noisy).The two values p(±1 | r k ) form the “intrinsic information” message ι k = {p 0k ,p 1k } derivedfrom each received symbol r k , 0 ≤ k ≤ 7. These values are computed from the receivedvector r =(0.1, −1.0, −0.7, 0.8, 1.1, 0.3, −0.9, 0.5) in the following table. (Note that itwould have sufficed to use the unnormalized pair {e r k /σ 2 ,e −r k /σ 2 }.) We also compute theHadamard transform {p 0k + p 1k =1,p 0k − p 1k = I k } of each pair of values for later use.1

k p 0k p 1k I kr 0 =+0.1 0.55 0.45 1 +0.10r 1 = −1.0 0.12 0.88 1 −0.76r 2 = −0.7 0.20 0.80 1 −0.60r 3 =+0.8 0.83 0.17 1 +0.66r 4 =+1.1 0.90 0.10 1 +0.80r 5 =+0.3 0.65 0.35 1 +0.30r 6 = −0.9 0.14 0.86 1 −0.72r 7 =+0.5 0.73 0.37 1 +0.46Note that I k =tanh r k /σ 2 ,so I k ≈ r k /σ 2 for small r k . The sign of I k represents a “harddecision,” whereas the magnitude 0 ≤ |I k |≤1 represents the reliability of that decision.We then propagate the APP messages through the graph below, using the BCJR (sumproduct)algorithm. Note that in the forward direction α 1 = ι 0 , α k is the sum-productupdate of α k−1 and ι k−1 for 2 ≤ k ≤ 7, and finally ε 7 = α 7 . Similarly, in the backwarddirection, β 7 = ι 7 , β k is the sum-product update of β k+1 and ι k for 6 ≥ k ≥ 1, and finallyε0 = β 1 .Then ε k is the sum-product update of α k and β k+1 for 1 ≤ k ≤ 6.Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7ι 0 ?6ε 0 ι ε 1 ε 2 ε 3 ε 4 ε 5 ε 6 εα 1 71?6ια 22?6ια 33?6ια 44?6ια 55?6ια 66?6ια 77?6- - - - - - - + + + + + + β 1 β 2 β 3 β 4 β 5 β 6 β 7For a (3, 2, 2) constraint code C k , the set of past 2-tuples consistent with a 0 value forany incident variable is C k (0) = {00, 11}, and similarly C k (1) = {01, 10}. Therefore thesum-product update rule isp(0 | r |P ) = p(0 | r |Pj )p(0 | r |Pj )+ p(1 | r |Pj )p(1 | r |Pj );p(1 | r |P ) = p(0 | r |Pj )p(1 | r |Pj )+ p(0 | r |Pj )p(1 | r |Pj ),where P j and P j are the two “pasts” upstream of P.Alternatively, following the hint, we may use the special rule for zero-sum nodes to obtainthe Hadamard transform of (p(0 | r |P ),p(1 | r |P )) simply by componentwise multiplicationof the Hadamard transforms of (p(0 | r |Pj ),p(1 | r |Pj )) and (p(0 | r |Pj ),p(1 | r |Pj )).By either method, we obtain the following values for the forward messages α k = {α 0k ,α 1k }and their Hadamard transforms {α 0k + α 1k =1,α 0k − α 1k = A k }, the backward messagesβk = {β 0k ,β 1k } and their Hadamard transforms {1,B k }, and the extrinsic informationmessages ε k = {ε 0k ,ε 1k } and their Hadamard transforms {1,E k }.α kA kα1 = ι 0 0.55 0.45 1 +0.10α 0k α 1kα2 0.46 0.54 1 −0.08α3 0.525 0.475 1 +0.05α4 0.515 0.485 1 +0.03α5 0.51 0.49 1 +0.02α6 0.5035 0.4965 1 +0.007α7 0.4975 0.5025 1 −0.0052

β k β 0k β 1k B kβ 7 = ι 7 0.73 0.27 1 +0.46β 6 0.335 0.665 1 −0.33β 5 0.45 0.55 1 −0.10β 4 0.46 0.54 1 −0.08β 3 0.475 0.525 1 −0.05β 2 0.515 0.485 1 +0.03β 1 0.49 0.51 1 −0.02ε k ε 0k ε 1k E kε 0 = β 1 0.49 0.51 1 −0.02ε 1 0.5015 0.4985 1 +0.003ε 2 0.5015 0.4985 1 +0.003ε 3 0.498 0.502 1 −0.004ε 4 0.4985 0.5015 1 −0.003ε 5 0.4965 0.5035 1 −0.007ε 6 0.5015 0.4985 1 +0.003ε 7 = α 7 0.4975 0.5025 1 −0.005Notice how the reliability of the forward and backward APP messages α k and β k degeneratesas more and more intrinsic information messages ι k are incorporated into them.The APP vectors {p(Y k =0 | r),p(Y k =1 | r)} for the symbol variables Y k are ultimatelyobtained by componentwise multiplication of ι k and ε k , normalized. We note that forall k, since ε 0k ≈ ε 1k ≈ 1 ,we have {p(Y 2 k =0 | r),p(Y k =1 | r)} ≈{ι 0k ,ι 1k }; i.e., theintrinsic information ι k dominates. Thus if hard decisions are made on each symbol, theresult is the same as if hard decisions had been made symbol-by-symbol based solely onthe channel outputs r k , and the resulting sequence of hard decisions is not a code sequencein the (8, 7, 2) code.In contrast, suppose that we perform the max-product (equivalent to the min-sum) algorithmwith this received sequence. We obtain the following values:α k α 0k α 1kα 1 = ι 0 0.55 0.45α 2 0.45 0.55α 3 0.55 0.45α 4 0.55 0.45α 5 0.55 0.45α 6 0.55 0.45α 7 0.45 0.553

β k β 0k β 1kβ 7 = ι 7 0.73 0.27β 6 0.27 0.73β 5 0.35 0.65β 4 0.35 0.65β 3 0.35 0.65β 2 0.65 0.35β 1 0.35 0.65ε k ε 0k ε 1kε 0 = β 1 0.35 0.65ε 1 0.55 0.45ε 2 0.55 0.45ε 3 0.45 0.55ε 4 0.45 0.55ε 5 0.45 0.55ε 6 0.55 0.45ε 7 = α 7 0.45 0.55In this case, the reliability of a forward or backward message is the minimum reliabilityof any intrinsic information that is incorporated into it. Eventually, this means that theextrinsic information ε 0 dominates the intrinsic information ι 0 for the least reliable symbolY 0 , so the original “hard decision” is “corrected” in this case. The same “correction” wouldbe performed by the Viterbi algorithm or by Wagner decoding.The max-product (min-sum) algorithm finds the maximum-likelihood code sequence,whereas the BCJR (sum-product) algorithm computes the APP vector of each bit. Abit decision based on the maximum APP minimizes the bit error probability, so the biterror probability could be (slightly) lower with the BCJR algorithm. ML sequence detectionminimizes the probability of decoding to the wrong codeword. The sequence ofmaximum-APP bits from the BCJR algorithm may not be a codeword, as we have justseen.Compare the complexity of BCJR decoding to Viterbi algorithm decoding (Problem 6.6).The BCJR algorithm is considerably more complex. For each trellis segment, it has tocompute the sum-product update rule three times. The straightforward sum-productupdate rule involves four multiplications and two additions; the simplified rule requires asingle multiplication. By contrast, the VA requires four additions and two comparisonsper trellis segment. Computation of the intrinsic APP vector is also more complex, andrequires estimation of the noise variance σ 2 , which could be a problem. Finally, the logicof the BCJR algorithm is more complex, since it is two-way rather than one-way.In short, the VA is significantly simpler than the BCJR algorithm. For this reason, the VAwas preferred for trellis decoding for many years. The BCJR algorithm was resurrectedwith the advent of turbo codes, where the fact that it produces “soft” (APP) outputs isessential to its role as a part of iterative decoding algorithms.(For this scenario, Wagner decoding is even simpler; see the solution to Problem 6.6.)4

Problem 9.1 (Iterative decoding on the BEC)(a) Using a graph of the (8, 4, 4) code like that of Figure 1 of Chapter 13 for iterativedecoding, decode the received sequence (1, 0, 0, ?, 0, ?, ?, ?).We may use the equivalent normal graph of Figure 3(b) of Chapter 11, which may beconverted to a directed normal graph (encoder) as shown in Figure 15 of Chapter 11,repeated below.y10 - = H HHHHHHHHHy01 - = XXX H HHHHHHHHH XX0XXy XX 2 - = Xjz- +H HHHHHHHHHy 9 13 = XXj- : X +X XXXX y04 - = XXz-XXX j +XXX XXX y 5 = 9 1 Xz- : + 1y6 = = y0 7 = The directed input edges show that (y 0 ,y 1 ,y 2 ,y 4 ) is an information set for this code. Given(y 0 ,y 1 ,y 2 ,y 4 )=(1, 0, 0, 0), the sum-product algorithm fills in the erasures by following thedirected state edges as shown above. On the first iteration, the first check yields y 3 =1;on the second iteration, the second and third checks yield y 5 = y 6 = 1; and on the thirdand last iteration, the last check yields y 7 =0.Then try to decode the received sequence (1, 1, 1, 1, ?, ?, ?, ?). Why does decoding fail in thelatter case? Give both a local answer (based on the graph) and a global answer (based onthe code).As shown in the graph below, iterative decoding stalls because at the initial iteration,the first check checks, and each of the last three checks has two or more erased inputs, sonone of the checks yields any new symbol values. The symbols (y 4 ,y 5 ,y 6 ,y 7 )and thelastthree checks are called a stopping set, because at least two of the symbols are involved ineach of the three checks; therefore if all are erased, no progress can be made.5

y10 - = H HHHHHHHHHy11 - = XX H HHHHHHHHH XXX1XXy XX 2 - = HXjz- +HHHHHHHHHy13 = 9XX : j- XXX +X XX y4 - = XX XX z- j XXX +XXXy5 = 9 XXz- : + y 6 = = y 7 = In this case, however, even a global (ML) decoder must fail, because there are two codewordswith (y 0 ,y 1 ,y 2 ,y 3 )=(1, 1, 1, 1), namely 11110000 and 11111111.(b) For the received sequence (1, 1, 1, 1, ?, ?, ?, 0), show that iterative decoding fails but thatglobal (i.e., ML) decoding succeeds.In this case, there is only one codeword with (y 0 ,y 1 ,y 2 ,y 3 ,y 7 )=(1, 1, 1, 1, 0), namely11110000, so global (ML) decoding will succeed.On the other hand, iterative decoding will fail, because the symbols (y 4 ,y 5 ,y 6 )and thelast three checks form a stopping set of this graph:: + y 4 - = X -XX +X XX XXy5 = 9 X z-: X + y 6 = = Problem 9.2 (Simulation of LDPC decoding on a BEC)(a) Perform a simulation of iterative decoding of a regular (d λ =3,d ρ =6) LDPC codeon a BEC with p = 0.45 (i.e., on Figure 9 of Chapter 13), and show how decoding getsstuck at the first fixed point (q l→r ≈ 0.35,q r→l ≈ 0.89). About how many iterations doesit take to get stuck?We start with q r→l = 1 and use the sum-product update relationships developed for thiscase in Chapter 13:q l→r =(0.45)(q r→l ) 2 ; q r→l =1 − (1 − q l→r ) 5 .6

We obtain the following sequence:q r→lq l→r1 0.450.949672 0.4058440.925954 0.3858260.912611 0.3747860.904469 0.3681290.899274 0.3639120.895868 0.3611610.893596 0.3593310.892064 0.3581000.891022 0.3572640.890311 0.3566940.889824 0.3563040.889490 0.3560360.889259 0.3558520.889101 0.3557250.888992 0.3556380.888916 0.3555770.888864 0.3555360.888828 0.3555070.888804 0.3554870.888787 0.3554740.888775 0.3554640.888767 0.3554580.888761 0.3554530.888757 0.3554500.888755 0.3554480.888753 0.3554470.888751 0.3554460.888751 0.3554450.888750 0.3554440.888750 0.3554440.888749 0.3554440.888749 0.355444... ...The point is that it takes a very finite number of iterations to get stuck, about 15–30 fulliterations, depending on how we define “stuck.”(b) By simulation of iterative decoding, compute the coordinates of the fixed point to sixsignificant digits.By the above simulation, the fixed point to six significant digits is at (q r→l ,q l→r ) =(0.888749, 0.355444).7

Problem 9.3 (Iterative decoding threshold)By analysis or simulation, show that the smallest p such that the equation x = 1−(1−px 2 ) 5∗has a solution in the interval 0 < x< 1 is p =0.429.... Explain the significance of thiscalculation for iterative decoding of LDPC codes on a BEC.A simulation such as that in Problem 9.2 will succeed for p =0.429 and get stuck for∗p = 0.430. Therefore p is somewhere between these two values.The significance of this calculation is that a rate-1/2 regular (d λ =3,d ρ = 6) LDPC code∗with iterative decoding can be used on any BEC with p

in,jin,j(b) Show that if we interchange w 0 and w 1 in an even number of incoming APPoutvectors, then the outgoing APP vector {w0 out ,w 1 } is unchanged. On the other hand,in,jshow that if we interchange w in,j 0 and w 1 in an odd number of incoming APP vectors,then the components w out out0 and w 1 of the outgoing APP vector are interchanged.in,jInterchanging w in,j and w 1 changes the sign of W in,j , while leaving W in,j 0 1 0 unchanged.Therefore if there an even number of such interchanges, W 0 out and W 1out are unchanged,out outand so therefore are w 0 and w 1 . If there an odd number of such interchanges, then thesign of W out is changed, which results in an interchange between w out out1 0 and w 1 .(c) Show that if we replace APP weight vectors {w 0 ,w 1 } by log likelihood ratios Λ =ln w 0 /w 1 , then the zero-sum sum-product update rule reduces to the “tanh rule” 1+ tanh Λ in,j /2outΛ =ln j,1 − tanh Λ in,j /2where the hyperbolic tangent is defined by tanh x = (e x − e −x )/(e x + e −x ).Since Λ in,j /2=ln(win,j/w in,j0 1) 1/2 ,wehave(w in,j /w in,j in,j /w in,j ) −1/2 in,j in,j− wtanh Λ in,j 0 1 ) 1/2 − (w/2= 0 1 w= 0 1= .(win,j in,j in,j/w/w in,j ) −1/2 in,j in,j+ w 1 W in,j0 1 ) 1/2 +(w 0 1 w 0 0 W in,j1 W 1outtanh Λ in,j /2=W = in,j W . outjin,j in,j in,j0 1 0W in,j1(Note that if (w in,j ,w ) are normalized so that w = W in,j0 + w 1 = 1, thentanh Λ in,j /2= W in,j 1 .) Consequentlyj j 0 0It follows that 1+ tanh Λ in,j /2 W out + W 1ln j0=ln=ln w 01 −j tanh Λin,j /2 W 0 out − W 1 out w 1out outout=Λ out .(d) Show that the “tanh rule” may alternatively be written astanh Λ out /2= tanh Λ in,j /2.As with the expression for tanh Λ in,j /2above,wehaveW 1outtanh Λ out /2= , Wout0whichwehaveshown abovetobeequal toj tanh Λin,j /2.j9