6.451 Principles of Digital Communication II Wednesday, May 4 ...
6.451 Principles of Digital Communication II Wednesday, May 4 ...
6.451 Principles of Digital Communication II Wednesday, May 4 ...
- No tags were found...
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
<strong>6.451</strong> <strong>Principles</strong> <strong>of</strong> <strong>Digital</strong> <strong>Communication</strong> <strong>II</strong> <strong>Wednesday</strong>, <strong>May</strong> 4, 2005MIT, Spring 2005 Handout #22Problem Set 9 SolutionsProblem 8.3 (revised) (BCJR (sum-product) decoding <strong>of</strong> SPC codes)As shown in Problem 6.4 or Figure 1 <strong>of</strong> Chapter 10, any (n, n − 1, 2) single-parity-checkcode has a two-state trellis diagram. Consider the (8, 7, 2) code, and let the receivedsequence from a discrete-time AWGN channel with input alphabet {±1} and noise variance2σ =1 be given by r =(0.1, −1.0, −0.7, 0.8, 1.1, 0.3, −0.9, 0.5). Perform BCJR (sumproduct)decoding <strong>of</strong> this sequence to determine the APP vector for each symbol Y k . [Hint:the special algorithm given in Section 13.5.2 (see Problem 9.5) to implement the sumproductupdate rule for zero-sum nodes may be helpful here.]First, let us draw the normal graph <strong>of</strong> a minimal trellis realization <strong>of</strong> the (8, 7, 2) code.This graph is shown abstractly below:Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7B0Σ1 BΣ 2 Σ 3 Σ 4 Σ 5 Σ 6 Σ 71 B 2 B 3 B 4 B 5 B 6 B 7(2, 1) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (3, 2) 1 (2, 1)Moreover, it is easy to see that the (3, 2) branch constraint codes are all (3, 2, 2) zero-sumcodes, and the (2, 1) codes are simple repetition codes that need not actually be shown.Therefore the trellis realization may be drawn simply as follows:Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7+ + + + + +Note that this trellis realization <strong>of</strong> the (8, 7, 2) code is another cycle-free realization thatuses 6 (3, 2, 2) zero-sum constraints, as in the reduced HT realization <strong>of</strong> Problem 8.1;however, the diameter <strong>of</strong> this realization is 5 (which is as large as it could possibly be).For a binary-input Gaussian-noise channel with inputs {±1} and Gaussian conditionalprobability density p(r | y) =(2πσ) −1/2 exp −(r − y) 2 /2σ 2 , the a posteriori probability(APP) <strong>of</strong> y ∈ {±1} given a received value r is, by Bayes’ rule,p(r | y)p(y | r) = p(r | 1) + p(r | −1)e ry/σ2= .er/σ 2 + e−r/σ 2Here σ 2 = 1 (so SNR = 1 (0 dB); i.e., the channel is very noisy).The two values p(±1 | r k ) form the “intrinsic information” message ι k = {p 0k ,p 1k } derivedfrom each received symbol r k , 0 ≤ k ≤ 7. These values are computed from the receivedvector r =(0.1, −1.0, −0.7, 0.8, 1.1, 0.3, −0.9, 0.5) in the following table. (Note that itwould have sufficed to use the unnormalized pair {e r k /σ 2 ,e −r k /σ 2 }.) We also compute theHadamard transform {p 0k + p 1k =1,p 0k − p 1k = I k } <strong>of</strong> each pair <strong>of</strong> values for later use.1
k p 0k p 1k I kr 0 =+0.1 0.55 0.45 1 +0.10r 1 = −1.0 0.12 0.88 1 −0.76r 2 = −0.7 0.20 0.80 1 −0.60r 3 =+0.8 0.83 0.17 1 +0.66r 4 =+1.1 0.90 0.10 1 +0.80r 5 =+0.3 0.65 0.35 1 +0.30r 6 = −0.9 0.14 0.86 1 −0.72r 7 =+0.5 0.73 0.37 1 +0.46Note that I k =tanh r k /σ 2 ,so I k ≈ r k /σ 2 for small r k . The sign <strong>of</strong> I k represents a “harddecision,” whereas the magnitude 0 ≤ |I k |≤1 represents the reliability <strong>of</strong> that decision.We then propagate the APP messages through the graph below, using the BCJR (sumproduct)algorithm. Note that in the forward direction α 1 = ι 0 , α k is the sum-productupdate <strong>of</strong> α k−1 and ι k−1 for 2 ≤ k ≤ 7, and finally ε 7 = α 7 . Similarly, in the backwarddirection, β 7 = ι 7 , β k is the sum-product update <strong>of</strong> β k+1 and ι k for 6 ≥ k ≥ 1, and finallyε0 = β 1 .Then ε k is the sum-product update <strong>of</strong> α k and β k+1 for 1 ≤ k ≤ 6.Y 0 Y 1 Y 2 Y 3 Y 4 Y 5 Y 6 Y 7ι 0 ?6ε 0 ι ε 1 ε 2 ε 3 ε 4 ε 5 ε 6 εα 1 71?6ια 22?6ια 33?6ια 44?6ια 55?6ια 66?6ια 77?6- - - - - - - + + + + + + β 1 β 2 β 3 β 4 β 5 β 6 β 7For a (3, 2, 2) constraint code C k , the set <strong>of</strong> past 2-tuples consistent with a 0 value forany incident variable is C k (0) = {00, 11}, and similarly C k (1) = {01, 10}. Therefore thesum-product update rule isp(0 | r |P ) = p(0 | r |Pj )p(0 | r |Pj )+ p(1 | r |Pj )p(1 | r |Pj );p(1 | r |P ) = p(0 | r |Pj )p(1 | r |Pj )+ p(0 | r |Pj )p(1 | r |Pj ),where P j and P j are the two “pasts” upstream <strong>of</strong> P.Alternatively, following the hint, we may use the special rule for zero-sum nodes to obtainthe Hadamard transform <strong>of</strong> (p(0 | r |P ),p(1 | r |P )) simply by componentwise multiplication<strong>of</strong> the Hadamard transforms <strong>of</strong> (p(0 | r |Pj ),p(1 | r |Pj )) and (p(0 | r |Pj ),p(1 | r |Pj )).By either method, we obtain the following values for the forward messages α k = {α 0k ,α 1k }and their Hadamard transforms {α 0k + α 1k =1,α 0k − α 1k = A k }, the backward messagesβk = {β 0k ,β 1k } and their Hadamard transforms {1,B k }, and the extrinsic informationmessages ε k = {ε 0k ,ε 1k } and their Hadamard transforms {1,E k }.α kA kα1 = ι 0 0.55 0.45 1 +0.10α 0k α 1kα2 0.46 0.54 1 −0.08α3 0.525 0.475 1 +0.05α4 0.515 0.485 1 +0.03α5 0.51 0.49 1 +0.02α6 0.5035 0.4965 1 +0.007α7 0.4975 0.5025 1 −0.0052
β k β 0k β 1k B kβ 7 = ι 7 0.73 0.27 1 +0.46β 6 0.335 0.665 1 −0.33β 5 0.45 0.55 1 −0.10β 4 0.46 0.54 1 −0.08β 3 0.475 0.525 1 −0.05β 2 0.515 0.485 1 +0.03β 1 0.49 0.51 1 −0.02ε k ε 0k ε 1k E kε 0 = β 1 0.49 0.51 1 −0.02ε 1 0.5015 0.4985 1 +0.003ε 2 0.5015 0.4985 1 +0.003ε 3 0.498 0.502 1 −0.004ε 4 0.4985 0.5015 1 −0.003ε 5 0.4965 0.5035 1 −0.007ε 6 0.5015 0.4985 1 +0.003ε 7 = α 7 0.4975 0.5025 1 −0.005Notice how the reliability <strong>of</strong> the forward and backward APP messages α k and β k degeneratesas more and more intrinsic information messages ι k are incorporated into them.The APP vectors {p(Y k =0 | r),p(Y k =1 | r)} for the symbol variables Y k are ultimatelyobtained by componentwise multiplication <strong>of</strong> ι k and ε k , normalized. We note that forall k, since ε 0k ≈ ε 1k ≈ 1 ,we have {p(Y 2 k =0 | r),p(Y k =1 | r)} ≈{ι 0k ,ι 1k }; i.e., theintrinsic information ι k dominates. Thus if hard decisions are made on each symbol, theresult is the same as if hard decisions had been made symbol-by-symbol based solely onthe channel outputs r k , and the resulting sequence <strong>of</strong> hard decisions is not a code sequencein the (8, 7, 2) code.In contrast, suppose that we perform the max-product (equivalent to the min-sum) algorithmwith this received sequence. We obtain the following values:α k α 0k α 1kα 1 = ι 0 0.55 0.45α 2 0.45 0.55α 3 0.55 0.45α 4 0.55 0.45α 5 0.55 0.45α 6 0.55 0.45α 7 0.45 0.553
β k β 0k β 1kβ 7 = ι 7 0.73 0.27β 6 0.27 0.73β 5 0.35 0.65β 4 0.35 0.65β 3 0.35 0.65β 2 0.65 0.35β 1 0.35 0.65ε k ε 0k ε 1kε 0 = β 1 0.35 0.65ε 1 0.55 0.45ε 2 0.55 0.45ε 3 0.45 0.55ε 4 0.45 0.55ε 5 0.45 0.55ε 6 0.55 0.45ε 7 = α 7 0.45 0.55In this case, the reliability <strong>of</strong> a forward or backward message is the minimum reliability<strong>of</strong> any intrinsic information that is incorporated into it. Eventually, this means that theextrinsic information ε 0 dominates the intrinsic information ι 0 for the least reliable symbolY 0 , so the original “hard decision” is “corrected” in this case. The same “correction” wouldbe performed by the Viterbi algorithm or by Wagner decoding.The max-product (min-sum) algorithm finds the maximum-likelihood code sequence,whereas the BCJR (sum-product) algorithm computes the APP vector <strong>of</strong> each bit. Abit decision based on the maximum APP minimizes the bit error probability, so the biterror probability could be (slightly) lower with the BCJR algorithm. ML sequence detectionminimizes the probability <strong>of</strong> decoding to the wrong codeword. The sequence <strong>of</strong>maximum-APP bits from the BCJR algorithm may not be a codeword, as we have justseen.Compare the complexity <strong>of</strong> BCJR decoding to Viterbi algorithm decoding (Problem 6.6).The BCJR algorithm is considerably more complex. For each trellis segment, it has tocompute the sum-product update rule three times. The straightforward sum-productupdate rule involves four multiplications and two additions; the simplified rule requires asingle multiplication. By contrast, the VA requires four additions and two comparisonsper trellis segment. Computation <strong>of</strong> the intrinsic APP vector is also more complex, andrequires estimation <strong>of</strong> the noise variance σ 2 , which could be a problem. Finally, the logic<strong>of</strong> the BCJR algorithm is more complex, since it is two-way rather than one-way.In short, the VA is significantly simpler than the BCJR algorithm. For this reason, the VAwas preferred for trellis decoding for many years. The BCJR algorithm was resurrectedwith the advent <strong>of</strong> turbo codes, where the fact that it produces “s<strong>of</strong>t” (APP) outputs isessential to its role as a part <strong>of</strong> iterative decoding algorithms.(For this scenario, Wagner decoding is even simpler; see the solution to Problem 6.6.)4
Problem 9.1 (Iterative decoding on the BEC)(a) Using a graph <strong>of</strong> the (8, 4, 4) code like that <strong>of</strong> Figure 1 <strong>of</strong> Chapter 13 for iterativedecoding, decode the received sequence (1, 0, 0, ?, 0, ?, ?, ?).We may use the equivalent normal graph <strong>of</strong> Figure 3(b) <strong>of</strong> Chapter 11, which may beconverted to a directed normal graph (encoder) as shown in Figure 15 <strong>of</strong> Chapter 11,repeated below.y10 - = H HHHHHHHHHy01 - = XXX H HHHHHHHHH XX0XXy XX 2 - = Xjz- +H HHHHHHHHHy 9 13 = XXj- : X +X XXXX y04 - = XXz-XXX j +XXX XXX y 5 = 9 1 Xz- : + 1y6 = = y0 7 = The directed input edges show that (y 0 ,y 1 ,y 2 ,y 4 ) is an information set for this code. Given(y 0 ,y 1 ,y 2 ,y 4 )=(1, 0, 0, 0), the sum-product algorithm fills in the erasures by following thedirected state edges as shown above. On the first iteration, the first check yields y 3 =1;on the second iteration, the second and third checks yield y 5 = y 6 = 1; and on the thirdand last iteration, the last check yields y 7 =0.Then try to decode the received sequence (1, 1, 1, 1, ?, ?, ?, ?). Why does decoding fail in thelatter case? Give both a local answer (based on the graph) and a global answer (based onthe code).As shown in the graph below, iterative decoding stalls because at the initial iteration,the first check checks, and each <strong>of</strong> the last three checks has two or more erased inputs, sonone <strong>of</strong> the checks yields any new symbol values. The symbols (y 4 ,y 5 ,y 6 ,y 7 )and thelastthree checks are called a stopping set, because at least two <strong>of</strong> the symbols are involved ineach <strong>of</strong> the three checks; therefore if all are erased, no progress can be made.5
y10 - = H HHHHHHHHHy11 - = XX H HHHHHHHHH XXX1XXy XX 2 - = HXjz- +HHHHHHHHHy13 = 9XX : j- XXX +X XX y4 - = XX XX z- j XXX +XXXy5 = 9 XXz- : + y 6 = = y 7 = In this case, however, even a global (ML) decoder must fail, because there are two codewordswith (y 0 ,y 1 ,y 2 ,y 3 )=(1, 1, 1, 1), namely 11110000 and 11111111.(b) For the received sequence (1, 1, 1, 1, ?, ?, ?, 0), show that iterative decoding fails but thatglobal (i.e., ML) decoding succeeds.In this case, there is only one codeword with (y 0 ,y 1 ,y 2 ,y 3 ,y 7 )=(1, 1, 1, 1, 0), namely11110000, so global (ML) decoding will succeed.On the other hand, iterative decoding will fail, because the symbols (y 4 ,y 5 ,y 6 )and thelast three checks form a stopping set <strong>of</strong> this graph:: + y 4 - = X -XX +X XX XXy5 = 9 X z-: X + y 6 = = Problem 9.2 (Simulation <strong>of</strong> LDPC decoding on a BEC)(a) Perform a simulation <strong>of</strong> iterative decoding <strong>of</strong> a regular (d λ =3,d ρ =6) LDPC codeon a BEC with p = 0.45 (i.e., on Figure 9 <strong>of</strong> Chapter 13), and show how decoding getsstuck at the first fixed point (q l→r ≈ 0.35,q r→l ≈ 0.89). About how many iterations doesit take to get stuck?We start with q r→l = 1 and use the sum-product update relationships developed for thiscase in Chapter 13:q l→r =(0.45)(q r→l ) 2 ; q r→l =1 − (1 − q l→r ) 5 .6
We obtain the following sequence:q r→lq l→r1 0.450.949672 0.4058440.925954 0.3858260.912611 0.3747860.904469 0.3681290.899274 0.3639120.895868 0.3611610.893596 0.3593310.892064 0.3581000.891022 0.3572640.890311 0.3566940.889824 0.3563040.889490 0.3560360.889259 0.3558520.889101 0.3557250.888992 0.3556380.888916 0.3555770.888864 0.3555360.888828 0.3555070.888804 0.3554870.888787 0.3554740.888775 0.3554640.888767 0.3554580.888761 0.3554530.888757 0.3554500.888755 0.3554480.888753 0.3554470.888751 0.3554460.888751 0.3554450.888750 0.3554440.888750 0.3554440.888749 0.3554440.888749 0.355444... ...The point is that it takes a very finite number <strong>of</strong> iterations to get stuck, about 15–30 fulliterations, depending on how we define “stuck.”(b) By simulation <strong>of</strong> iterative decoding, compute the coordinates <strong>of</strong> the fixed point to sixsignificant digits.By the above simulation, the fixed point to six significant digits is at (q r→l ,q l→r ) =(0.888749, 0.355444).7
Problem 9.3 (Iterative decoding threshold)By analysis or simulation, show that the smallest p such that the equation x = 1−(1−px 2 ) 5∗has a solution in the interval 0 < x< 1 is p =0.429.... Explain the significance <strong>of</strong> thiscalculation for iterative decoding <strong>of</strong> LDPC codes on a BEC.A simulation such as that in Problem 9.2 will succeed for p =0.429 and get stuck for∗p = 0.430. Therefore p is somewhere between these two values.The significance <strong>of</strong> this calculation is that a rate-1/2 regular (d λ =3,d ρ = 6) LDPC code∗with iterative decoding can be used on any BEC with p
in,jin,j(b) Show that if we interchange w 0 and w 1 in an even number <strong>of</strong> incoming APPoutvectors, then the outgoing APP vector {w0 out ,w 1 } is unchanged. On the other hand,in,jshow that if we interchange w in,j 0 and w 1 in an odd number <strong>of</strong> incoming APP vectors,then the components w out out0 and w 1 <strong>of</strong> the outgoing APP vector are interchanged.in,jInterchanging w in,j and w 1 changes the sign <strong>of</strong> W in,j , while leaving W in,j 0 1 0 unchanged.Therefore if there an even number <strong>of</strong> such interchanges, W 0 out and W 1out are unchanged,out outand so therefore are w 0 and w 1 . If there an odd number <strong>of</strong> such interchanges, then thesign <strong>of</strong> W out is changed, which results in an interchange between w out out1 0 and w 1 .(c) Show that if we replace APP weight vectors {w 0 ,w 1 } by log likelihood ratios Λ =ln w 0 /w 1 , then the zero-sum sum-product update rule reduces to the “tanh rule” 1+ tanh Λ in,j /2outΛ =ln j,1 − tanh Λ in,j /2where the hyperbolic tangent is defined by tanh x = (e x − e −x )/(e x + e −x ).Since Λ in,j /2=ln(win,j/w in,j0 1) 1/2 ,wehave(w in,j /w in,j in,j /w in,j ) −1/2 in,j in,j− wtanh Λ in,j 0 1 ) 1/2 − (w/2= 0 1 w= 0 1= .(win,j in,j in,j/w/w in,j ) −1/2 in,j in,j+ w 1 W in,j0 1 ) 1/2 +(w 0 1 w 0 0 W in,j1 W 1outtanh Λ in,j /2=W = in,j W . outjin,j in,j in,j0 1 0W in,j1(Note that if (w in,j ,w ) are normalized so that w = W in,j0 + w 1 = 1, thentanh Λ in,j /2= W in,j 1 .) Consequentlyj j 0 0It follows that 1+ tanh Λ in,j /2 W out + W 1ln j0=ln=ln w 01 −j tanh Λin,j /2 W 0 out − W 1 out w 1out outout=Λ out .(d) Show that the “tanh rule” may alternatively be written astanh Λ out /2= tanh Λ in,j /2.As with the expression for tanh Λ in,j /2above,wehaveW 1outtanh Λ out /2= , Wout0whichwehaveshown abovetobeequal toj tanh Λin,j /2.j9