The Effect of Spatial Coupling on Compressive Sensing
Abstract
Recently, it was observed that spatiallycoupled LDPC code ensembles approach the Shannon capacity for a class of binaryinput memoryless symmetric (BMS) channels. The fundamental reason for this was attributed to a threshold saturation phenomena derived in [1]. In particular, it was shown that the belief propagation (BP) threshold of the spatially coupled codes is equal to the maximum a posteriori (MAP) decoding threshold of the underlying constituent codes. In this sense, the BP threshold is saturated to its maximum value. Moreover, it has been empirically observed that the same phenomena also occurs when transmitting over more general classes of BMS channels.
In this paper, we show that the effect of spatial coupling is not restricted to the realm of channel coding. The effect of coupling also manifests itself in compressed sensing. Specifically, we show that spatiallycoupled measurement matrices have an improved sparsity to sampling threshold for reconstruction algorithms based on verification decoding. For BPbased reconstruction algorithms, this phenomenon is also tested empirically via simulation. At the block lengths accessible via simulation, the effect is quite small and it seems that spatial coupling is not providing the gains one might expect. Based on the threshold analysis, however, we believe this warrants further study.
I Introduction
This work investigates the effect of spatial coupling in compressed sensing. Spatiallycoupled codes are a class of protographbased lowdensity paritycheck (LDPC) codes capable of achieving near capacity performance, under lowcomplexity belief propagation (BP) decoding, when transmitting over binaryinput memoryless symmetric (BMS) channels. The history of these codes can be traced back to the work of Felström and Zigangirov [2], where they were introduced as convolutional LDPC code ensembles. There is a considerable literature on convolutionallike LDPC ensembles. Variations in their constructions as well as some analysis can be found in [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15].
The fundamental reason underlying the remarkable performance was recently discussed in detail in [1] for the case where the transmission takes place over the binary erasure channel. Before we go further, we briefly explain the construction of spatiallycoupled codes; for more details, see [1].
Ia Spatially coupled codes – () ensemble
Recall that a regular LDPC code ensemble can be represented by the protograph (or base graph) as shown in Fig. 1. Spatially coupled code ensemble, denoted by , is constructed by considering a protograph created by taking multiple copies of the protograph (see the figure on the righthandside in Fig. 1) and connecting them as shown in the Fig. 2. We stress here that this is only the protograph. The code is constructed by taking multiple copies of this base graph and interconnecting them using a random permutation.
For the sake of exposition we consider regular LDPC ensemble, with odd and . However, coupled codes can also be constructed starting from any standard ()regular LDPC code ensemble [1]. To achieve the coupling, connect each protograph to protographs “to the left” and to protographs “to the right.” This is shown in Figure 2 for the case and . An extra check nodes are added on each side to connect the “overhanging” edges at the boundary.
There are two main effects resulting from this coupling (see [1] for details):
(i) Rate Reduction:
Recall that the design rate of the underlying standard regular ensemble is .
The design rate of the corresponding
ensemble, due to boundary effects, is reduced to
(1) 
(ii) Threshold Increase: Let , and denote the threshold^{1}^{1}1The BP(MAP) threshold, of a fixed ensemble of codes, denotes the channel parameter value below which the BP(MAP) decoder succeeds and fails above it. of the BP decoder for the ensemble, MAP threshold of the ensemble and the MAP threshold of the underlying ensemble, respectively. Then the main result of [1] is that, when we transmit spatially coupled codes over the BEC we have,
See [1] for a precise statement of the main theorem. The effect of coupling can been nicely seen by plotting the EXIT curves for the uncoupled and coupled codes. This is shown in Fig. 3. Similar phenomena can also be empirically observed when transmitting over more general BMS channels [16, 17].
IB Outline
In this work, we study the effect of spatial coupling in the problem of compressed sensing. We begin with our compressive sensing setup and explain our decoders in the next section. In the same section, we introduce spatiallycoupled measurement matrices. We then develop the density evolution (DE) equations for the class of decoders which we consider. In Section III we perform experiments depicting the effect of spatial coupling. We conclude with a short discussion of interesting open questions.
Ii Compressed Sensing
Compressed sensing (CS) is now one of the most exciting new areas in signal processing. It is based on the idea that many realworld signals (e.g., those sparse in some transform domain) can be reconstructed from a small number of linear measurements. This idea originated in the areas of statistics and signal processing [18, 19, 20, 21, 22], but is also quite related to previous work in computer science [23, 24, 25] and applied mathematics [26, 27, 28]. CS is also very closely related to errorcorrecting codes, and can be seen as source coding using linear codes over real numbers [29, 30, 31, 32, 33, 34, 35, 36].
The basic problem is to measure and then reconstruct a signal vector using the linear observation model , where the matrix is the measurement matrix and is a noise vector. The signal vector is called sparse if it contains at most nonzero entries. If, instead of being identically zero, the other entries are much smaller than the largest entries, then the vector is called approximately sparse.
The main goal of this paper is to study the effect of spatial coupling on the performance of CS systems. Spatial coupling has been shown to drastically increase the threshold LDPC codes with an asymptotically negligible decrease in the code rate. The tight connection between lowdensity paritycheck (LDPC) codes and CS means that one would expect spatial coupling to improve messagepassing decoders for CS systems. A more subtle question is whether or not traditional CS decoders based on convex relaxations (e.g., basis pursuit and LASSO) will also benefit from spatial coupling. In this paper, we compare various standard constructions of measurement matrices with constructions based on spatial coupling. These tests are performed using a variety of CS decoders and a few signal models.
Iia System Model
All our CS results rely on sparse measurement matrices. A variety of such matrices have been considered before in [29, 32, 30, 31, 37, 38]. Their main advantage is that they enable a variety of lowcomplexity reconstruction techniques. In general, the entries of the measurement matrix are either chosen to be plus/minus one with equal probability, or drawn from a continuous distribution. The latter provides some benefit when the signal contains only a small set of nonzero values.
Throughout the paper we will consider two kinds of measurement matrices. The first type of measurement matrix is generated from the paritycheck matrix sampled uniformly at random from the ensemble of a regular LDPC codes. The sampling ratio, , is given by . The second type of measurement matrix, which we call a spatially coupled measurement matrix, will be generated from the paritycheck matrix sampled from the ensemble. In this case, the sampling ratio can be obtained from (1) as . For finite , the number of measurements of the spatially coupled measurement matrix is larger than the corresponding uncoupled matrix. But as , the two measurement matrices have an equal number of measurements.
The signal is assumed to be drawn i.i.d. from some distribution . The parameter determines the fraction nonzero entries in the signal. To model sparse vectors, one typically assumes that has a delta function at with mass . For approximately sparse vectors, we use the twoGaussian mixture model from [32, 39]. In this model, all entries are independent zeromean Gaussians but a fraction have variance and a fraction have variance (with ).
IiB Message Passing Reconstruction
Messagepassing (MP) reconstruction for compressedsensing systems based on sparse measurement matrices was introduced by Sarvotham, Baron, and Baraniuk in [29, 32, 39]. The tight connection between CS and errorcorrecting codes enabled researchers to quickly analyze other MP reconstruction schemes based on tools from modern coding theory [30, 31, 37, 33]. Recently, there has also been some progress in analyzing the performance of these schemes for approximately sparse signals and noisy observations [39, 35, 40].
For sparse measurement matrices, the asymptotic performance of MP reconstruction can be analyzed (in theory) using density evolution (DE) [41]. Indeed, this works well for simplified suboptimal reconstruction algorithms like the ”sudocodes” reconstruction [29, 42]. For true beliefpropagation (BP) reconstruction, however, numerical evaluation of the DE equations is intractable. This means that it is difficult to determine the asymptotic behavior of a particular sparse ensemble with BP reconstruction. Recently, Donoho, Maleki and Montanari [35], [34] proposed an approximate message passing (AMP) algorithm for compressed sensing with dense Gaussian measurement matrices. For this algorithm, they introduced a variant of DE (known as state evolution) that provides a precise characterization of its performance.
IiC Analytical Setup
Our analytical results rely on the suboptimal reconstruction technique introduced for ”sudocodes” by Sarvotham, Baron, and Baraniuk [29]. The ”sudocodes” reconstruction technique falls into the class of verification decoders that was introduced and analyzed by Luby and Mitzenmacher for LDPC codes over large alphabets [43]. In this paper, we use the messagepassing based implementation of the second (more powerful) algorithm from their paper and refer to it as LM2 [44]. The main drawback of this choice is that the analysis only works for strictly sparse vectors where the measurements are observed without noise. The main benefit is that one can analyze its performance precisely using density evolution (DE) and construct EXITlike curves to illustrate the benefits of spatial coupling. For example, in the large system limit, this leads to provable sparsity thresholds where reconstruction succeeds with probability one [31, 42].
Although the LM2 decoder is unstable in the presence of noise, this does not mean that its threshold is meaningless in practice. The LM2 decoder can be seen as a suboptimal version of listmessage passing (LMP) [44] which itself can be seen as a limiting case of the full beliefpropagation (BP) decoder for CS [32, 39]. Ideally, one would analyze the BP decoder directly, but performing a DE analysis for decoders which pass real functions as messages [32, 39] remains intractable. Still, we expect that a complete analysis of the BP decoder would show that its expected performance is always better than the LM2 decoder and that the BP decoder allows stable reconstruction below its sparsity threshold.
Verification decoding rules for messagebased LM2 decoding in a CS system: (Check Node) The output message on an edge: equals the unique value which satisfies the observation constraint based on all other input edges; this output message is verified if and only if all other input messages are verified. (Symbol Node) The output message is: verified and equal to the input message if any input message is verified, verified and equal to 0 if any input message on another edge is equal to 0, verified and equal to the matching value if any two other input messages match, and unverified with a value of 0 otherwise.
Initially, the check nodes with measurement equal to zero, transmit zero on all their outgoing edges. This arises from the basic property of verification decoders when we consider the nonzero values to come from a continuous distribution. The scheme described above does not guarantee that all verified symbols are actually correct. The event that a symbol is verified but incorrect is called false verification (FV). If either the nonzero matrix entries, or the signal values, are drawn from a continuous distribution, then the FV event has probability zero.
IiC1 Protograph Density Evolution for LM2 Decoding
For generality, we consider a verificationbased decoding rules for channels with erasures and errors. The variables will be used to denote, respectively, the probability that the message type is erasure (E), incorrect (I), correct and unverified (C), and verified (V). The DE equations for LM2 decoding of the standard irregular LDPC code ensemble are given in [43, p. 125]. For the protograph setup (including erasures), we derive the DE equations below.
For a check node of degree , let the messagetype probability for the th input edge and be the messagetype probability for the th output edge. Then, we have the update rules
(2) 
In words, these four disjoint probabilities are for the events: “at least one E”, “at least one E or I minus at least E”, “all C or V minus all V”, and “all V”.
For a bit node of degree , let the messagetype probability for the th input edge and be the messagetype probability for the th output edge. If is the probability of channel error and is the probability of channel erasure, then we have the update rules
(3) 
Let be the event that “all input edges are E or I except for at most one C”. In words, these four disjoint probabilities are for the events: “channel erased and ”, “channel error and ”, “channel correct and all input edges are E or I”, and “not or channel correct and ”.
We can specialize the above DE equations to the case when we do not have spatial coupling. More precisely, consider the measurement matrix generated by the regular LDPC ensemble. Then for the check node we have,
(4) 
and for the variable node side we have,
(5) 
These equations can be compared to [43, p. 125] by setting , , mapping , and mapping . We remark here that the channel error probability, , is equal to the fraction of nonzero symbols in the signal.
IiC2 EXITlike curves
In theory of iterative codes, the EXIT function is defined by [45], where denotes the vector of all observations except . In words, it is the uncertainty in decoding a bit, when its channel observation is discarded. In general one can replace by , where is the extrinsic estimate of any other decoder (e.g., BP decoder). Such EXITlike curves are useful visualization tool in iterative coding theory. They have also been used in iterative coding theory to provide deep results relating the BP decoder and the optimal MAP decoder [45]. It is hence of great interest to visualize our results by plotting EXITlike curves even in the case of compressed sensing.
According to the rules of LM2 decoder, a variable node is verified, or its value is perfectly known, when the variable node receives either two or more (correct but unverified) messages or one or more (verified) message. In this case the EXIT value of the variable node is zero. Thus the EXIT value of a variable node is proportional to the probability that a variable node is unverified. In other words, it receives “all I (incorrect) except at most one C (correct but unverified)”. This is exactly the probability of the event A (cf. Section IIC). To summarize: in our experiments we plot an EXITlike curve which is the probability of a variable node being unverified when we change the sparsity ratio continuously.
Iii Experiments
Iiia Verification Decoders
We perform DE analysis of the verification decoder. More precisely, we consider the DE equations (IIC1) to (IIC1). In particular we consider the case when , i.e., we have corruption of symbols only via errors in the channel.
IiiA1 EXITlike curves for fixed sampling ratio
We first consider measurement matrices generated by the regular LDPC code. This fixes the sampling ratio to . We then fix the sparsity ratio and run DE equations (IIC1) and (IIC1), till a fixed point is reached. We then use the fixed point densities to evaluate the probability that a particular node is unverified. We plot this value on the vertical axis in Fig. 4, for different values of . In this case, the EXITlike curve is illustrated by the light gray curve (leftmost) in Fig. 4. We observe that below , the fixed point is trivial. More precisely, for sparsity ratio , the probability that a node is unverified goes to zero. This means that, with high probability, the LM2 decoder is able to reconstruct the signal exactly. For , we see that the LM2 decoder fails to reconstruct the signal and the probability of a variable node being unverified is nonzero.
Similar experiment is now done with spatially coupled measurement matrices. We run the DE equations given by (IIC1) and (IIC1) for different lengths, . For , the threshold is . The reason for such a large value of the threshold is the because the sampling ratio is much larger than (cf. equation (1)). As increases the curves move to the left, which is similar to the effect observed in channel coding (cf. Fig. 3). As increases, the resultant measurement matrix resembles more and more like the uncoupled measurement matrix (locally) and the sampling ratio also approaches . However, the curves seem to get stuck at . To summarize: for large we have the sampling ratio very close to and we observe that the sparsity threshold is much larger, .
IiiA2 Phase transition
We run the DE equations for different values of the sampling ratio. Let us explain this more precisely. As usual, we consider an uncoupled () measurement matrix generated and its coupled version, . We fix and consider for different values of . As a consequence, we obtain different values of . We run DE equations (IIC1) and (IIC1) for and for fixed for all the experiments.
Figure 5 shows the results. The light circles correspond to the regular case and the dark circles correspond to the coupled case. We see that for the sampling ratios considered, the coupled measurement matrices have an improved sparsitysampling tradeoff. At the first, the trend of the circles may seem a bit strange. But, the normalized coordinates of the plot imply that a family of systems achieving a fixed oversampling ratio family would give a horizontal line.
The continuous curve shown is Donoho’s phase transition for weak CS reconstruction using LP [46]. This was later identified in [35] as the LP threshold for AMP reconstruction of a sparse signal with a worstcase amplitude distribution. For this reason, the comparison is not really fair. The LM2 thresholds are for the noiseless measurement of a special class of signals, while LP reconstruction thresholds are for the stable reconstruction of a random signal drawn from any i.i.d. distribution. Still, this result does highlight the fact that better performance is possible (and might even be achieved by LP reconstruction) in some special cases.
IiiB BP and LP Reconstruction
Simulations were also performed for beliefpropagation (BP) and linearprogramming (LP) reconstruction of coupled and uncoupled measurement matrices. The initial goal was to choose system parameters that would allow direct comparison with [39, Fig. 3]. Unfortunately, the rate loss associated with coupled codes made it difficult to achieve fair comparisons in this case. Therefore, the tested system uses noiseless measurements to reconstruct signals from the twoGaussian model with , , , and . For the coupled and uncoupled systems, measurement matrices are based on regular codes with and . Nonzero entries in these matrices are randomly chosen to be either or . These parameters give asymptotic sampling ratios of , but the coupled systems have slightly more measurements due to finite rate loss.
For each setup, the experiment tested a single (randomly constructed) measurement matrix on the same set of 10 randomly generated signal vectors with and (the number of nonzero symbols in the signal). During each trial, the BP decoder is run for 30 iterations and the rootmeansquare error (RMSE) is calculated. The curves in Figure 6 show the median RMSE (over 10 trials) for the particular parameters.
In Fig. 6 we compare the coupled and uncoupled measurement matrix for increasing average degrees. In Fig. 7 we compare the performance of CSBP and LP decoders when we consider coupled and uncoupled measurement matrix with variable node degree fixed to 7. The figure also shows the performance curve when we use the Sarvotham, Barron, Baraniuk (SBB) (see [39]) ensemble for measuring.
We remark here that the simulation setup for this section uses between 24 and 48. The number of coupling stages must be kept somewhat low for two reasons: (i) to prevent short cycles in the decoding graph and (ii) to reduce the decoding time as the number of required decoding iterations increases with . These small values of result in a slightly larger sampling ratio (i.e., more measurements) for the coupled measurement matrices as opposed to uncoupled measurement matrices. This effect is handled correctly on the simulation curves, but makes direct comparison somewhat difficult. As a consequence, we observe that there is no appreciable gain by using couplng in the case when we are using the CSBP decoder. The performance changes are very small and do not support a conclusion that spatialcoupling provides large gains for CS with BP reconstruction.
The basispursuit LP reconstruction technique was also tested with each measurement matrix and signal vector. The results for symbol degree are shown in Figure 7. From this, one observes that LP reconstruction benefits even less from spatial coupling.
However, results from the channel coding setup imply that the performance for moderate is very close to the performance when is very large. For example, Figures 3 and 4 show that the threshold for has already saturated to the threshold. Nevertheless, experiments with larger values of merit further investigation.
Iv Discussion
Recently, it was shown in [47] that the effect of coupling is also observed in many other problems, like the satisfiability, graph coloring and CurieWeiss model of statistical physics. In this paper, we found that spatiallycoupled measurement matrices have an improved sparsitysampling ratio for model used by verification decoding. This provides evidence that the phenomena of spatial coupling is quite general. On the other hand, we also observed that spatiallycoupled measurement matrices provide little (if any) gains for the compressedsensing problem with moderate blocklengths and beliefpropagation reconstruction.
We conclude with possible future research directions.

It would be interesting to see the effect of spatial coupling on other reconstruction techniques for compressed sensing. For example, Donoho, Maleki and Montanari recently proposed an approximate message passing (AMP) algorithm for compressed sensing. They showed that the AMP, when tuned to the minimax thresholding function, achieves the same sparsity to sampling tradeoff as decoding. It would be interesting to investigate the effect of spatial coupling on AMP, especially if it can be tuned to minimization for .

There are also interesting open questions regarding EXITlike curves for compressedsensing reconstruction. What is the meaning, if there is any, of the limiting curve in Figure 4? Can we define an EXITlike curve for CS reconstruction that obeys an Area Theorem like in the case of channel coding [45].
V Acknowledgments
SK acknowledges support of NMC via the NSF collaborative grant CCF0829945 on “Harnessing Statistical Physics for Computing and Communications”. The work of HP was supported by the National Science Foundation under Grant No. CCF0747470. The authors would also like to thank Misha Chertkov, Andrea Montanari, Rüdiger Urbanke for many useful discussions and encouragement.
References
 [1] S. Kudekar, T. Richardson, and R. Urbanke, “Threshold saturation via spatial coupling: Why convolutional LDPC ensembles perform so well over the BEC,” 2010, eprint: http://arxiv.org/abs/1001.1826.
 [2] A. J. Felström and K. S. Zigangirov, “Timevarying periodic convolutional codes with lowdensity paritycheck matrix,” IEEE Trans. Inform. Theory, vol. 45, no. 5, pp. 2181–2190, Sept. 1999.
 [3] K. Engdahl and K. S. Zigangirov, “On the theory of low density convolutional codes I,” Problemy Peredachi Informatsii, vol. 35, no. 4, pp. 295–310, 1999.
 [4] K. Engdahl, M. Lentmaier, and K. S. Zigangirov, “On the theory of lowdensity convolutional codes,” in AAECC13: Proceedings of the 13th International Symposium on Applied Algebra, Algebraic Algorithms and ErrorCorrecting Codes. London, UK: SpringerVerlag, 1999, pp. 77–86.
 [5] M. Lentmaier, D. V. Truhachev, and K. S. Zigangirov, “To the theory of lowdensity convolutional codes. II,” Probl. Inf. Transm., vol. 37, no. 4, pp. 288–306, 2001.
 [6] R. M. Tanner, D. Sridhara, A. Sridharan, T. E. Fuja, and D. J. Costello, Jr., “LDPC block and convolutional codes based on circulant matrices,” IEEE Trans. Inform. Theory, vol. 50, no. 12, pp. 2966 – 2984, Dec. 2004.
 [7] A. Sridharan, M. Lentmaier, D. J. Costello, Jr., and K. S. Zigangirov, “Convergence analysis of a class of LDPC convolutional codes for the erasure channel,” in Proc. of the Allerton Conf. on Commun., Control, and Computing, Monticello, IL, USA, Oct. 2004.
 [8] M. Lentmaier, A. Sridharan, K. S. Zigangirov, and D. J. Costello, Jr., “Terminated LDPC convolutional codes with thresholds close to capacity,” in Proc. of the IEEE Int. Symposium on Inform. Theory, Adelaide, Australia, Sept. 2005.
 [9] ——, “Iterative decoding threshold analysis for LDPC convolutional codes,” IEEE Trans. Info. Theory, Oct. 2010.
 [10] D. G. M. Mitchell, A. E. Pusane, K. S. Zigangirov, and D. J. Costello, Jr., “Asymptotically good LDPC convolutional codes based on protographs,” in Proc. of the IEEE Int. Symposium on Inform. Theory, Toronto, CA, July 2008, pp. 1030 – 1034.
 [11] M. Lentmaier, G. P. Fettweis, K. S. Zigangirov, and D. J. Costello, Jr., “Approaching capacity with asymptotically regular LDPC codes,” in Information Theory and Applications, San Diego, USA, Feb. 8–Feb. 13, 2009, pp. 173–177.
 [12] R. Smarandache, A. Pusane, P. Vontobel, and D. J. Costello, Jr., “Pseudocodewords in LDPC convolutional codes,” in Proc. of the IEEE Int. Symposium on Inform. Theory, Seattle, WA, USA, July 2006, pp. 1364 – 1368.
 [13] ——, “Pseudocodeword performance analysis for LDPC convolutional codes,” IEEE Trans. Inform. Theory, vol. 55, no. 6, pp. 2577–2598, June 2009.
 [14] M. Papaleo, A. Iyengar, P. Siegel, J. Wolf, and G. Corazza, “Windowed erasure decoding of LDPC convolutional codes,” in Proc. of the IEEE Inform. Theory Workshop, Cairo, Egypt, Jan. 2010, pp. 78 – 82.
 [15] M. Lentmaier and G. P. Fettweis, “On the thresholds of generalized LDPC convolutional codes based on protographs,” in Proc. of the IEEE Int. Symposium on Inform. Theory, Austin, USA, 2010.
 [16] S. Kudekar, C. Méasson, T. Richardson, and R. Urbanke, “Threshold saturation on BMS channels via spatial coupling,” Sept. 2010, 6th International Symposium on Turbo Codes and Iterative Information Processing.
 [17] M. Lentmaier, D. G. M. Mitchell, G. P. Fettweis, and D. J. Costello, Jr., “Asymptotically good LDPC convolutional codes with AWGN channel thresholds close to the Shannon limit,” Sept. 2010, 6th International Symposium on Turbo Codes and Iterative Information Processing.
 [18] S. Chen, D. Donoho, and M. Saunders, “Atomic decomposition by basis pursuit,” SIAM J. Sci. Comp., vol. 20, no. 1, pp. 33–61, 1998.
 [19] I. Gorodnitsky and B. Rao, “Sparse signal reconstruction from limited data using FOCUSS: A reweighted minimum norm algorithm,” Signal Processing, IEEE Transactions on, vol. 45, no. 3, pp. 600–616, 2002.
 [20] S. Cotter, B. Rao, K. Engan, and K. KreutzDelgado, “Sparse solutions to linear inverse problems with multiple measurement vectors,” IEEE Trans. Signal Processing, vol. 53, no. 7, pp. 2477–2488, 2005.
 [21] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory, vol. 52, no. 4, pp. 1289–1306, 2006.
 [22] E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inform. Theory, vol. 52, no. 2, pp. 489–509, 2006.
 [23] A. Gilbert, S. Muthukrishnan, and M. Strauss, “Approximation of functions over redundant dictionaries using coherence.” SODA, 2003.
 [24] G. Cormode and S. Muthukrishnan, “An improved data stream summary: the countmin sketch and its applications,” Journal of Algorithms, vol. 55, no. 1, p. 75, 2005.
 [25] A. C. Gilbert, M. J. Strauss, J. A. Tropp, and R. Vershynin, “One sketch for all: Fast algorithms for compressed sensing,” in in Proceedings of the ACM Symposium on the Theory of Computing (STOC 2007), 2007.
 [26] A. Cohen, W. Dahmen, and R. DeVore, “Compressed sensing and best kterm approximation,” RWTHAachen IGPM, Tech. Rep., July 2006.
 [27] W. Johnson and J. Lindenstrauss, “Extensions of lipschitz maps into hilbert space,” Contemp. Math., vol. 26, pp. 189–206, 1984.
 [28] E. D. Gluskin, “Norms of random matrices and widths of finitedimensional sets,” Math. USSR Sbornik, vol. 48, pp. 173–182, 1984.
 [29] S. Sarvotham, D. Baron, and R. G. Baraniuk, “Sudocodes–fast measurement and reconstruction of sparse signals,” pp. 2804–2808, July 2006.
 [30] W. Xu and B. Hassibi, “Efficient compressive sensing with deterministic guarantees using expander graphs,” in Proc. IEEE Inform. Theory Workshop, Lake Tahoe, CA, Sept. 2007, pp. 414–419.
 [31] F. Zhang and H. D. Pfister, “Compressed sensing and linear codes over real numbers,” in Proc. 3rd Annual Workshop on Inform. Theory and its Appl., San Diego, CA, Feb. 2008.
 [32] S. Sarvotham, D. Baron, and R. Baraniuk, “Compressed sensing reconstruction via belief propagation,” Rice University, Tech. Rep. ECE0601, July 2006.
 [33] W. Dai and O. Milenkovic, “Subspace pursuit for compressive sensing signal reconstruction,” 2008. [Online]. Available: http://arxiv.org/abs/0803.0811
 [34] D. Donoho, A. Maleki, and A. Montanari, “Messagepassing algorithms for compressed sensing: I. motivation and construction,” in Proc. IEEE Inform. Theory Workshop, Cairo, Egypt, Jan. 2010.
 [35] ——, “Messagepassing algorithms for compressed sensing,” Proc. Natl. Acad. Sci. USA, vol. 106, no. 45, pp. 18 914–18 919, 2009.
 [36] ——, “Online supplement to messagepassing algorithms for compressed sensing,” Proc. Natl. Acad. Sci. USA, vol. 106, no. 45, 2009, [Online]. DOI: 10.1073/pnas.0909892106.
 [37] F. Zhang and H. D. Pfister, “On the iterative decoding of high rate LDPC codes with applications in compressed sensing,” in Proc. 47th Annual Allerton Conf. on Commun., Control, and Comp., Monticello, IL, Sept. 2008.
 [38] R. Berinde, A. Gilbert, P. Indyk, H. Karloff, and M. Strauss, “Combining geometry and combinatorics: A unified approach to sparse signal recovery,” 2008, pp. 798–805.
 [39] D. Baron, S. Sarvotham, and R. Baraniuk, “Bayesian compressive sensing via belief propagation,” IEEE Trans. Signal Processing, vol. 58, no. 1, pp. 269–280, 2010.
 [40] M. Bayati and A. Montanari, “The dynamics of message passing on dense graphs, with applications to compressed sensing,” 2010, [Online]. Available: http://arxiv.org/abs/1001.3448.
 [41] T. Richardson, M. A. Shokrollahi, and R. Urbanke, “Design of capacityapproaching irregular lowdensity paritycheck codes,” IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb. 2001.
 [42] F. Zhang and H. D. Pfister, “On the iterative decoding of high rate LDPC codes with applications in compressed sensing,” 2009, submitted to IEEE Trans. on Inform. Theory also available in Arxiv preprint cs.IT/0903.2232.
 [43] M. Luby and M. Mitzenmacher, “Verificationbased decoding for packetbased lowdensity paritycheck codes,” IEEE Trans. Inform. Theory, vol. 51, no. 1, pp. 120–127, 2005.
 [44] F. Zhang and H. D. Pfister, “Listmessage passing achieves capacity on the ary symmetric channel for large ,” 2008, submitted to IEEE Trans. on Inform. Theory also available in Arxiv preprint cs.IT/0806.3243.
 [45] T. Richardson and R. Urbanke, Modern Coding Theory. Cambridge University Press, 2008.
 [46] D. Donoho, “Highdimensional centrally symmetric polytopes with neighborliness proportional to dimension,” Discrete and Computational Geometry, vol. 35, no. 4, pp. 617–652, 2006.
 [47] S. H. Hassani, N. Macris, and R. Urbanke, “Coupled graphical models and their thresholds,” in Proc. of the IEEE Inform. Theory Workshop, Dublin, Ireland, Sept. 2010.