# Spatially-Coupled MacKay-Neal Codes and Hsu-Anastasopoulos Codes

xx \noxx \fieldA \SpecialSectionInformation Theory and Its Applications \authorlist \authorentry[kenta@comm.ss.titech.ac.jp]Kenta KASAImTITECH \authorentry[sakaniwa@ss.titech.ac.jp]Kohichi SAKANIWAfTITECH \affiliate[TITECH]Dept. of Communications and Integrated Systems, Tokyo Institute of Technology 2-12-1, O-Okayama, Meguro-ku, Tokyo, 152-8550, Japan {summary} Kudekar et al. recently proved that for transmission over the binary erasure channel (BEC), spatial coupling of LDPC codes increases the BP threshold of the coupled ensemble to the MAP threshold of the underlying LDPC codes. One major drawback of the capacity-achieving spatially-coupled LDPC codes is that one needs to increase the column and row weight of parity-check matrices of the underlying LDPC codes.

It is proved, that Hsu-Anastasopoulos (HA) codes and MacKay-Neal (MN) codes achieve the capacity of memoryless binary-input symmetric-output channels under MAP decoding with bounded column and row weight of the parity-check matrices. The HA codes and the MN codes are dual codes each other.

The aim of this paper is to present an empirical evidence that spatially-coupled MN (resp. HA) codes with bounded column and row weight achieve the capacity of the BEC. To this end, we introduce a spatial coupling scheme of MN (resp. HA) codes. By density evolution analysis, we will show that the resulting spatially-coupled MN (resp. HA) codes have the BP threshold close to the Shannon limit. patial coupling, LDPC code, iterative decoding

## 1 Introduction

Achieving the capacity of memoryless binary-input symmetric-output (MBS) channels [1, Chap. 4] under efficient decoding algorithms used to be an ultimate goal for coding theorists. The capacity of MBS channels was practically achieved by irregular LDPC codes [1] with belief propagation (BP) decoding and rigorously achieved by polar codes [2] with successive cancellation. Spatially-coupled (SC) LDPC codes [3] have bounded range of parity-check constrains like convolutional codes. Recently, SC-LDPC codes have attracted much attention due to the fact that the codes achieves the capacity of binary erasure channels (BEC) [3] and an observation that the codes seem to achieve the capacity of MBS channels [4].

SC-LDPC codes are capacity-achieving codes designed based on the construction of convolutional LDPC codes. Felström and Zigangirov [5] introduced a construction method of -regular convolutional LDPC codes from -regular block LDPC codes [1]. The convolutional LDPC codes exhibited better decoding performance than the underlying block LDPC codes under a fair comparison with respect to the code length. Note that in this paper, convolutional LDPC codes are defined by sparse band parity-check matrices. Lentmaier et al. observed that (4,8)-regular convolutional LDPC codes exhibited the decoding performance surpassing the BP threshold of (4,8)-regular block LDPC codes [6]. Sridharan et al. developed the density evolution (DE) [1] for the BEC and calculated the BP threshold [7]. Lentmaier et al. developed the DE for the binary-input memoryless (BMS) channels [8]. Further, the BP threshold equals to the MAP threshold of the underlying block LDPC codes with a lot of accuracy. The MAP threshold was calculated by the extended BP (EBP) extrinsic information transfer (EXIT) function analysis [9]. Constructing convolutional LDPC codes from a block LDPC code improves the BP threshold up to the MAP threshold of the underlying codes.

Kudekar et al. named this phenomenon “threshold saturation” and proved rigorously for the BEC [3]. In the limit of large and , the SC-LDPC code ensemble [3] was shown to achieve the MAP threshold of -regular LDPC code ensemble. The parameters and are the coupling number and the randomized window size. For more details, we refer the readers to [3].

Further, by computing EBP generalized EXIT (GEXIT) curves [9], Kudekar et al. [4] observed empirical evidence which supports the threshold saturation occurs also for the BMS channels. For arbitrary BMS channels, the MAP threshold of the codes quickly converge to the Shannon limit while the BP threshold goes to 0 [10]. In other words, in the limit of large as keeping , and , the SC-LDPC code ensemble achieves universally the capacity of the BMS channels under BP decoding. Such universality is not supported by other efficiently-decodable capacity-achieving codes, i.e., polar codes [2] and irregular LDPC codes [11]. According to the channel, polar codes need selection of frozen bits [12] and irregular LDPC codes need optimization of degree distributions.

One major drawback of the code ensemble is that one needs to increase degree and to strictly achieve the capacity. The number of non-zero entries in the parity-check matrices is proportional to the decoding computations of BP decoding. Specifically, for the BEC case, this is identical to the total computations for decoding [13]. The number of edges per information bit is called density. For example the density of -regular LDPC codes is . For fixed , the density is unbounded as . It is desired to reduce the density. Lentmaier et al. constructed coupled Accumulate-Repeat-Jagged-Accumulate (ARJA) codes [14] which exhibit very good BP threshold. The density of the ARJA code is bounded but the BP threshold leaves a small gap to the Shannon limit.

For any code achieving a fraction of capacity of the BMS under MAP decoding, Sason and Urbanke showed that the density of the code is at least , where and are constant [15]. In other words, the density of capacity-achieving codes needs to be unbounded. However, this is not the case for the codes with puncture. The punctured bits work as auxiliary states and help decoding. Such puncturing is widely used for structured codes [16]. Pfister and Sason constructed Accumulate-Repeat-Accumulate (ARA) codes and Accumulate-LDPC (ALDPC) codes which achieve the capacity of BEC with bounded density [17].

MacKay-Neal codes [18] are non-systematic two-edge type LDPC codes [16, 1]. The MacKay-Neal (MN) codes are conjectured to achieve the capacity of BMS channels under ML decoding. Murayama et al. [19] and Tanaka et al. [20] reported the empirical evidence of the conjecture for BSC and AWGN channels, respectively by a non-rigorous statistical mechanics approach known as replica method.

Hsu and Anastasopoulos [21] rigorously proved that LDPC codes concatenated with LDGM (low-density generator-matrix) codes [22] achieve the capacity of arbitrary BMS channels with bounded density under ML decoding. We name the codes Hsu-Anastasopoulos (HA) codes after the inventors. Furthermore, Wainwright and Martinian showed HA codes achieve the rate-distortion bound for symmetric Bernoulli sources [23].

On the contrary to their ML decoding performance, it is interesting to see that the MN and HA codes have no BP thresholds. In other words, the bit error rate under BP decoding does not go to 0, even if none of the transmitted bits are erased.

The aim of this paper is to present an empirical evidence that SC-MN (resp. SC-HA) codes with bounded density achieve the capacity of the BEC. To this end, we introduce a spatial coupling scheme of MN (resp. HA) codes. Spatial coupling of MN and HA codes has never been studied before. By DE analysis, we will show that the resulting SC-MN (resp. SC-HA) codes have the BP threshold close to the Shannon limit.

## 2 Preliminaries

In this section, we briefly review the SC-LDPC codes introduced by Kudekar et al. [3] and their results of performance analysis. We assume For simplicity, we focus on rate 1/2 codes, i.e., .

The SC-LDPC codes are defined by the following protograph codes [24]. The adjacency matrix of the protograph is referred to as a base matrix. The base matrix of () SC-LDPC code is given as follow. Let us define as . Let be an band binary matrix of band size and column weight , where the band size is height width of the band. For example is given in Fig. 2. The SC-LDPC codes is defined as protograph codes defined by base matrix . Figure 1 shows protograph of SC-LDPC codes. The protograph of SC-LDPC codes have variable nodes and check nodes. Hence, the design coding rate of SC-LDPC codes is given by

which converges to as with vanishing gap like

Table 2 shows BP threshold values of SC-LDPC codes. As increases, it is observed that the design coding rate converges to 1/2 and the BP threshold values approach the MAP threshold value of LDPC codes .

Kudekar et al. observed does not converge to . There remains a small gap of order in significant digits. In order to decrease the gap, Kudekar et al. introduced the SC-LDPC codes that allows randomized connection of edges of window size [3]. It is shown that converges to as and tend to infinity as

It is known [3] that quickly converges to the Shannon limit , i.e.,

Table 1 shows the MAP threshold values of LDPC codes.

(3,6) | 0.4294 | 0.48815 |

(4,8) | 0.3834 | 0.49774 |

(5,10) | 0.3416 | 0.49949 |

(6,12) | 0.3075 | 0.49988 |

(7,14) | 0.2797 | 0.49997 |

This implies that SC-LDPC codes achieve the capacity of the BEC, in the limit of large column and row weight. On the other hand, infinite column and row weight are required for the codes to achieve the capacity.

1 | 0.714309 | 0.166667 |
---|---|---|

2 | 0.587842 | 0.300000 |

4 | 0.512034 | 0.388889 |

8 | 0.488757 | 0.441176 |

16 | 0.488151 | 0.469697 |

32 | 0.488151 | 0.484615 |

64 | 0.488151 | 0.492248 |

128 | 0.488151 | 0.496109 |

## 3 Bounded-Density Codes Achieving Capacity under ML Decoding

In this section, we give the definition of MN codes and HA codes.

### 3.1 MacKay-Neal codes

Let be a random binary matrix of size with column weight and row weight . Let be a random binary matrix of size with column weight and row weight .

An -MN code is defined as an LDPC code with parity-check matrix

(1) |

with and such that the bits corresponding to are punctured by the non-systematic fashion of the codes. Sparse parity-check representation is given by , with information bits as state bits and parity bits . From , it follows that the generator matrix of the -MN code is given by

(2) |

The MN codes are non-systematic, in other words, only the parity bits are transmitted through the channel.

The -MN codes are called non-systematic Repeat-Accumulate (RA) codes in [17]. The -MN codes are identical to ()-LDGM codes.

We give another definition of MN codes. Both definitions are equivalent in terms of DE. The MN codes can be defined by a multi-edge type LDPC code ensemble [16] with degree distribution pair

The design coding rate of an -MN code is given by .

Murayama et al. [19] and Tanaka et al. [20] reported empirical evidences that MN codes achieve the capacity of BSC and AWGN channels under ML decoding, respectively by a non-rigorous statistical mechanics approach known as replica method. From those results, we expect that MN codes achieve the capacity of arbitrary MBS channel if respectively under ML decoding. Hence, we will use .

Let and be the erasure probability of messages from information and parity bit nodes in the -th round of BP decoding, respectively. DE gives the update equations of density and as follows.

It is obvious that for for any . It follows does not converge to 0 as , even if . Hence, MN codes have no BP threshold.

### 3.2 Hsu-Anastasopoulos Codes

An -HA code is a concatenation of an -LDPC code and a -LDGM code. Let be a random binary matrix of size with column weight and row weight . Let be a random binary matrix of size with column weight and row weight . An -HA code is defined as an LDPC code with parity-check matrix

(3) |

with and such that the bits corresponding to and are punctured. It follows that the parity-check matrix of the -HA code is given by

(4) |

From Eqs. (2) and (4) it follows that if we set , , and , it holds . It follows the -HA code is dual code of the -MN code.

The HA codes can be defined by a multi-edge type LDPC code ensemble [16] with degree distribution pair

The design coding rate of an -HA code is given by . The -HA code is referred to an accumulate LDPC code in [17]. The -HA code is identical to an -regular LDPC code. Sparse parity-check representation is given by and , with state bits and parity bits .

For the arbitrary MBS channels, it is shown that -HA codes achieve the capacity under ML decoding with bounded [21].

## 4 Spatially-Coupled MN and HA Codes

### 4.1 Spatially-Coupled MacKay-Neal Codes

In this section, we propose spatial coupling of MN codes and evaluate the BP threshold values. For simplicity we focus on MN codes with , where . We define SC-MN codes as follows. First, let be a binary band matrix of band size and column weight . For example is given in Fig. 2. Next, let be a binary band matrix of band size and column weight . For example, is given in Fig. 2. An SC-MN code is defined as a protograph code which is defined by base matrix

(5) |

and such that the bits corresponding to are punctured. Equations (1) and (5) are analogous in such a way that both have the same column-weight and almost the same row-weight distributions, and one is sparse matrix and the other is sparse band matrix under some permutation of rows and columns.

In Fig. 3, we show a protograph of SC-MN codes.

In the protograph of SC-MN codes, there are punctured variable nodes unpunctured variable nodes, and check nodes. Hence, in the limit of large , the design coding rate is given as

(6) |

The density is given by

The minimum density is attained when and .

Table 3 shows the BP threshold values and rates of SC-MN codes. As increasing , it is observed that approaches a value close to 1/2. From Eq. (6), it holds that This observation supports that threshold saturation occurs by SC-MN codes.

2 | 0.561146 | 0.363636 |
---|---|---|

4 | 0.511397 | 0.421053 |

8 | 0.500252 | 0.457143 |

16 | 0.499977 | 0.477612 |

32 | 0.499908 | 0.488550 |

### 4.2 Spatially-Coupled Hsu-Anastasopoulos Codes

In this section, we propose spatial coupling of HA codes and evaluate the BP threshold values. An SC-HA code is defined as a protograph code which is defined by base matrix

and such that the bits corresponding to the left sub-matrix are punctured. Figure 4 shows the protograph SC-HA codes.

We assume that The protograph of the SC-HA codes has punctured information bit node, unpunctured bit nodes, and check nodes. The design coding rate is given by

(7) |

The density is given by

The minimum density is attained when and .

Table 4 shows the BP threshold values of SC-HA code and design coding rates. As increasing , it is observed that approach a value close to 1/2. From Eq. (7), it holds that This observation supports that threshold saturation occurs by SC-HA codes.

1 | 0.695420 | 0.285714 |
---|---|---|

2 | 0.594441 | 0.363636 |

4 | 0.516970 | 0.421053 |

8 | 0.500460 | 0.457143 |

16 | 0.499980 | 0.477612 |

32 | 0.499909 | 0.488550 |

Discussion: We have proposed spatial-coupling of MN codes and HA codes. It is observed that the BP threshold values for BEC are very close to the Shannon limit. In other words, threshold saturation occurs for SC-MN codes and SC-HA codes. We observed such threshold saturation for various parameters with . However, these BP threshold values leave a small gap to the Shannon limit. Such a gap was also observed when evaluating the BP threshold of the SC-LDPC codes [3]. Kudekar et al. proved that this is explained by wiggles appearing in EBP EXIT curves [3]. In order to decrease the gap, they introduced randomized SC-LDPC codes that allow connections of edges with window size . We observed the same effect on wiggles at the EBP EXIT curve of SC-MN codes. We will discuss the above observation in the next section.

## 5 Randomized Spatially-Coupled MacKay-Neal Codes

In this section, we consider randomized SC-MN codes. Randomized SC-HA codes can be considered in a similar way. For simplicity, we focus only on randomized SC-MN codes.

We define an SC-MN code ensemble as follows. The Tanner graph of a code in the SC-MN code ensemble is constructed as follows. At each section , consider information nodes of degree , parity nodes of degree , and check nodes which has incident information nodes and parity nodes. Connect randomly these nodes in such a way that for and , information nodes at section and check nodes at section are connected with edges and parity nodes at section and check nodes at section are connected with edges. Shorten the information and parity bits at section , i.e., set the bits to zero and do not transmit them. Puncture the information nodes at section , i.e., the bits are not transmitted. Note that this ensemble is nicely represented by joint degree distributions [25]. The definition of the SC-MN code ensemble is based on that of randomized SC-LDPC code ensemble. For more details on randomized SC-LDPC code ensemble, we refer the readers to [3, Section II.B].

Denote the number of transmitted bit nodes, punctured bit nodes by , , respectively.

The number of check nodes of degree at least 1, denoted by , can be counted by the same way as in [3, Lemma 3] as follows.

The design coding rate is given by

Let and be the erasure probability of messages emitting from information-bit and parity-bit nodes, respectively, at section at the -th round of BP decoding in the limit of large . DE update equations of the randomized SC-MN code are given as follows. For , for . For , for . For ,

Consider fixed points of the DE system, i.e., such that for and

For any , a fixed point is called trivial. Trivial fixed points is corresponds to the message density of successful decoding. The EBP EXIT curve is defined as the projected plots of fixed points , other than trivial ones, onto the following EXIT function.

Figure 5 (a) plots the EBP EXIT curve of the SC-MN code ensemble. Consider no points of the EBP curve are at . This suggests that there are only trivial fixed points of DE with and converges to the trivial fixed point. It follows that the BP threshold is given by at which the left-most cliff edge of the curve vertically drops [3]. The BP threshold in Fig. 5 (a) is very close to 1/2. However, there exists a small gap. A closer look at the cliff edge of the green curve given in Fig. 5 (b) reveals the cliff of SC-MN codes have wiggles. The wiggle size decreases by increasing as seen in Fig. 5 (b).

Figure 6 and 7 plot the EBP EXIT curves and of the SC-MN code ensemble for and , respectively. It is observed that as decreasing value, starts to collapse from the boundaries and gradually to the center . It is observed that uncollapsed takes almost the same value . Each time collapse, is affected a little, which gives rise to a wiggle. By increasing , each collapses slowly, which gives rise to a smaller wiggle.

## 6 Conclusion

We have proposed spatial-coupling of MN (resp. HA) codes with bounded density. By DE analysis, we observed that the BP threshold values of the SC-MN (resp. SC-HA) codes for BEC are very close to the Shannon limit. This empirical evidence supports that the SC-MN (resp. SC-HA) codes may achieve the Shannon limit of the BEC. In other words, threshold saturation occurs for SC-MN codes and SC-HA codes. We observed such threshold saturation for various parameters with .

We further investigate the EBP EXIT curve of randomized SC-MN codes. We observed that the small gap between BP threshold and the Shannon limit is caused by wiggles of BP EXIT curve. By increasing the randomized window size , the wiggle size largely decreased.

## Acknowledgment

The first author would like to thank, chronologically, Tadashi Wadayama, Michael Lentmaier, Rüdiger Urbanke, David Saad, Ryuhei Mori, Kazushi Mimura, Hironori Uchikawa, Shrinivas Kudekar, Toshiyuki Tanaka, and Yoshiyuki Kabashima for their help and useful discussions. He is also grateful to the associate editor and the anonymous reviewers for their helpful and constructive comments.

## References

- [1] T. Richardson and R. Urbanke, Modern Coding Theory, Cambridge University Press, March 2008.
- [2] E. Arıkan, “Channel polarization: A method for constructing capacity-achieving codes for symmetric binary-input memoryless channels,” IEEE Trans. Inf. Theory, vol.55, no.7, pp.3051–3073, July 2009.
- [3] S. Kudekar, T. Richardson, and R. Urbanke, “Threshold saturation via spatial coupling: Why convolutional LDPC ensembles perform so well over the BEC,” IEEE Trans. Inf. Theory, vol.57, no.2, pp.803–834, Feb. 2011.
- [4] S. Kudekar, C. Méasson, T.J. Richardson, and R.L. Urbanke, “Threshold saturation on BMS channels via spatial coupling,” http://arxiv.org/abs/1004.3742, April 2010.
- [5] A.J. Felström and K.S. Zigangirov, “Time-varying periodic convolutional codes with low-density parity-check matrix,” IEEE Trans. Inf. Theory, vol.45, no.6, pp.2181–2191, June 1999.
- [6] M. Lentmaier, D.V. Truhachev, and K.S. Zigangirov, “To the theory of low-density convolutional codes. II,” Probl. Inf. Transm. , no.4, pp.288–306, 2001.
- [7] A. Sridharan, J. M. Lentmaier, D. J. Costello, and K.S. Zigangirov, “Convergence analysis of a class of LDPC convolutional codes for the erasure channel,” Proc. 42th Annual Allerton Conf. on Commun., Control and Computing, Montecillo, Illinois, USA, Oct. 2004.
- [8] M. Lentmaier, A. Sridharan, D.J. Costello, Jr., and K.S. Zigangirov, “Iterative decoding threshold analysis for LDPC convolutional codes,” IEEE Trans. Inf. Theory, vol.56, no.10, pp.5274 –5289, Oct. 2010.
- [9] C. Méasson, A. Montanari, T. Richardson, and R. Urbanke, “The generalized area theorem and some of its consequences,” IEEE Trans. Inf. Theory, vol.55, no.11, pp.4793 –4821, Nov. 2009.
- [10] G. Miller and D. Burshtein, “Bounds on the maximum-likelihood decoding error probability of low-density parity-check codes,” IEEE Trans. Inf. Theory, vol.47, no.7, pp.2696–2710, Nov. 2001.
- [11] T.J. Richardson, M.A. Shokrollahi, and R.L. Urbanke, “Design of capacity-approaching irregular low-density parity-check codes,” IEEE Trans. Inf. Theory, vol.47, no.2, pp.619–637, Feb. 2001.
- [12] R. Mori and T. Tanaka, “Performance and construction of polar codes on symmetric binary-input memoryless channels,” http://arxiv.org/abs/0901.2207, Jan. 2009.
- [13] A. Khandekar and R. McEliece, “On the complexity of reliable communication on the erasure channel,” Proc. 2001 IEEE Int. Symp. Inf. Theory (ISIT), p.1, June 2001.
- [14] D. Mitchell, M. Lentmaier, and D. Costello, “New families of ldpc block codes formed by terminating irregular protograph-based LDPC convolutional codes,” Proc. 2010 IEEE Int. Symp. Inf. Theory (ISIT), pp.824 –828, June 2010.
- [15] I. Sason and R. Urbanke, “Parity-check density versus performance of binary linear block codes over memoryless symmetric channels,” IEEE Trans. Inf. Theory, vol.49, no.7, pp.1611 – 1635, July 2003.
- [16] T. Richardson and R. Urbanke, “Multi-edge type LDPC codes,” 2003. preprint available at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.106.7310.
- [17] H. Pfister and I. Sason, “Accumulate repeat accumulate codes: Capacity-achieving ensembles of systematic codes for the erasure channel with bounded complexity,” IEEE Trans. Inf. Theory, vol.53, no.6, pp.2088 –2115, June 2007.
- [18] D. MacKay, “Good error-correcting codes based on very sparse matrices,” IEEE Trans. Inf. Theory, vol.45, no.2, pp.399 –431, March 1999.
- [19] T. Murayama, Y. Kabashima, D. Saad, and R. Vicente, “Statistical physics of regular low-density parity-check error-correcting codes,” Phys. Rev. E, vol.62, no.2, pp.1577–1591, Aug. 2000.
- [20] T. Tanaka and D. Saad, “Typical performance of regular low-density parity-check codes over general symmetric channels,” J. Phys. A: Math. Gen. , vol.36, no.43, pp.11143–11157, Oct. 2003.
- [21] C.H. Hsu and A. Anastasopoulos, “Capacity-achieving codes with bounded graphical complexity and maximum likelihood decoding,” IEEE Trans. Inf. Theory, vol.56, no.3, pp.992 –1006, March 2010.
- [22] J. Cheng and R. McEliece, “Some high-rate near capacity codecs for the gaussian channel,” Proc. 34th Annual Allerton Conf. on Commun., Control and Computing, Oct. 1996.
- [23] M. Wainwright and E. Martinian, “Low-density graph codes that are optimal for binning and coding with side information,” IEEE Trans. Inf. Theory, vol.55, no.3, pp.1061 –1079, March 2009.
- [24] J. Thorpe, “Low-density parity-check (LDPC) codes constructed from protographs,” IPN Progress Report, pp.42–154, Aug. 2003.
- [25] K. Kasai, T. Shibuya, and K. Sakaniwa, “Detailedly represented irregular low-density parity-check codes,” IEICE Trans. Fundamentals, vol.E86-A, no.10, pp.2435–2444, Oct. 2003.