Random subdictionaries and coherence conditions for sparse signal recovery
The most frequently used condition for sampling matrices employed in compressive sampling is the restricted isometry (RIP) property of the matrix when restricted to sparse signals. At the same time, imposing this condition makes it difficult to find explicit matrices that support recovery of signals from sketches of the optimal (smallest possible) dimension. A number of attempts have been made to relax or replace the RIP property in sparse recovery algorithms. We focus on the relaxation under which the near-isometry property holds for most rather than for all submatrices of the sampling matrix, known as statistical RIP or StRIP condition. We show that sampling matrices of dimensions with maximum coherence and mean square coherence support stable recovery of -sparse signals using Basis Pursuit. These assumptions are satisfied in many examples. As a result, we are able to construct sampling matrices that support recovery with low error for sparsity higher than which exceeds the range of parameters of the known classes of RIP matrices.
One of the important problems in theory of compressed sampling is construction of sampling operators that support algorithmic procedures of sparse recovery. A universal sufficient condition for stable reconstruction is given by the restricted isometry property (RIP) of sampling matrices . It has been shown that sparse signals compressed to low-dimensional images using linear RIP maps can be reconstructed using minimization procedures such as Basis pursuit and Lasso [20, 18, 14, 11],
Let be an -dimensional real signal that has a sparse representation in a suitably chosen basis. We will assume that has nonzero coordinates (it is a -sparse vector) or is approximately sparse in the sense that it has at most significant coordinates, i.e., entries of large magnitude compared to the other entries. The observation vector is formed as a linear transformation of , i.e.,
where is an real matrix, and is a noise vector. We assume that has bounded energy (i.e., ). The objective of the estimator is to find a good approximation of the signal after observing . This is obviously impossible for general signals but becomes tractable if we seek a sparse approximation which satisfies
for some and constants . Note that if itself is -sparse, then (1) implies that the recovery error is at most proportional to the norm of the noise. Moreover it implies that the recovery is stable in the sense that if is approximately -sparse then the recovery error is small. If the estimate satisfies an inequality of the type (1), we say that the recovery procedure satisfies a error guarantee.
Among the most studied estimators is the Basis Pursuit algorithm . This is an -minimization algorithm that provides an estimate of the signal through solving a convex programming problem
Basis Pursuit is known to provide both and error guarantees under the conditions on discussed in the next section.
Another popular estimator for which the recovery guarantees are proved using coherence properties of the sampling matrix is Lasso [45, 23]. Assume the vector is independent of the signal and formed of independent identically distributed Gaussian random variables with zero mean and variance Lasso is a regularization of the minimization problem written as follows:
Here is a regularization parameter which controls the complexity (sparsity) of the optimizer.
Compressed sensing is just one of a large group of applications of solutions of severely ill-defined problems under the sparsity assumption. An extensive recent overview of such applications is given in . It is this multitude of concrete applications that makes the study of sparse recovery such an appealing area of signal processing and applied statistics.
1.1. Properties of sampling matrices
One of the main questions related to sparse recovery is derivation of sufficient conditions for the convergence and error guarantees of the reconstruction algorithms. Here we discuss some properties of sampling matrices that are relevant to our results, focusing on incoherence and near-isometry of random submatrices of the sampling matrix.
Let be an real matrix and let be its columns. Without loss of generality throughout this paper we assume that the columns are unit-length vectors. Let and let be a -subset of the set of coordinates. By we denote the set of all -subsets of Below we write to refer to the submatrix of formed of the columns with indices in . Given a vector we denote by a -dimensional vector given by the projection of the vector on the coordinates in .
It is known that at least samples are required for any recovery algorithm with an error guarantee of the form (1) (see for example [36, 37]). Matrices with random Gaussian or Bernoulli entries with high probability provide the best known error guarantees from the sketch dimension that matches this lower bound [20, 21, 19]. The estimates become more conservative once we try to construct sampling matrices explicitly.
We say that satisfies the coherence property if the inner product is uniformly small, and call the coherence parameter of the matrix. The importance of incoherent dictionaries has been recognized in a large number of papers on compressed sensing, among them [46, 49, 30, 17, 15, 16, 11]. The coherence condition plays an essential role in proofs of recovery guarantees in these and many other studies. We also define the mean square coherence and the maximum average square coherence of the dictionary:
Of course, with equality if and only if for every the sum in takes the same value. Our reliance on two coherence parameters of the sampling matrix resembles somewhat the approach in [3, 4]; however, unlike those papers, our results imply recovery guarantees for Basis Pursuit. Our proof methods are also materially different from these works. More details are provided below in this section where we comment on previous results.
The RIP property
The matrix satisfies the RIP property (is -RIP) if
holds for all -sparse vectors , where is a parameter. Equivalently, is -RIP if holds for all , where is the spectral norm and Id is the identity matrix. The RIP property provides a sufficient condition for the solution of (2) to satisfy the error guarantees of Basis Pursuit [20, 18, 14, 11]. In particular, by , -RIP suffices for both and error estimates, while  improves this to -RIP.
As is well known (see  ), coherence and RIP are related: a matrix with coherence parameter is -RIP. This connection has served the starting point in a number of studies on constructing RIP matrices from incoherent dictionaries. To implement this idea one starts with a set of unit vectors with maximum coherence In other words, we seek a well-separated collection of lines through the origin in , or reformulating again, a good packing of the real projective space One way of constructing such packings begins with taking a set of binary -dimensional vectors whose pairwise Hamming distances are concentrated around Call the maximum deviation from the width of the set An incoherent dictionary is obtained by mapping the bits of a small-width code to bipolar signals and normalizing. The resulting coherence and width are related by
One of the first papers to put forward the idea of constructing RIP matrices from binary vectors was the work by DeVore . While  did not make a connection to error-correcting codes, a number of later papers pursued both its algorithmic and constructive aspects [6, 12, 13, 24]. Examples of codes with small width are given in , where they are studied under the name of small-bias probability spaces. RIP matrices obtained from the constructions in  satisfy . Ben-Aroya and Ta-Shma  recently improved this to for The advantage of obtaining RIP matrices from binary or spherical codes is low construction complexity: in many instances it is possible to define the matrix using only columns while the remaining columns can be computed as their linear combinations. We also note a result by Bourgain et al.  who gave the first (and the only known) construction of RIP matrices with on the order of (i.e., greater than ). An overview of the state of the art in the construction of RIP matrices is given in a recent paper .
At the same time, in practical problems we still need to write out the entire matrix; so constructions of complexity are an acceptable choice. Under these assumptions, the best tradeoff between and for RIP-matrices based on codes and coherence is obtained from Gilbert-Varshamov type code constructions: namely, it is possible to construct -RIP matrices with . At the same time, already  observes that the sketch dimension in RIP matrices constructed from binary codes is at least
Statistical incoherence properties
The limitations on incoherent dictionaries discussed in the previous section suggest relaxing the RIP condition. An intuitively appealing idea is to require that condition (4) hold for almost all rather than all -subsets replacing RIP with a version of it, in which the near-isometry property holds with high probability with respect to the choice of Statistical RIP (StRIP) matrices are arguably easier to construct, so they have a potential of supporting provable recovery guarantees from shorter sketches compared to the known constructive schemes relying on RIP.
A few words on notation. Let and let denote the set of -subsets of The usual notation for probability is used to refer a probability measure when there is no ambiguity. At the same time, we use separate notation for some frequently encountered probability spaces. In particular, we use to denote the uniform probability distribution on . If we need to choose a random -subset and a random index in we use the notation . We use to denote any probability measure on which assigns equal probability to each of the orthants (i.e., with uniformly distributed signs).
An matrix satisfies the statistical RIP property (is -StRIP) if
In other words, the inequality
holds for at least a proportion of all -subsets of and for all
A related but different definition was given later in several papers such as [12, 3, 30] as well as some others. In these works, a matrix is called -StRIP if inequality (5) holds for at least proportion of -sparse unit vectors . While several well-known classes of matrices were shown to have this property, it is not sufficient for sparse recovery procedures. Several additional properties as well as specialized recovery procedures that make signal reconstruction possible were investigated in .
In this paper we focus on the statistical isometry property as given by Def. 1 and mean this definition whenever we mention StRIP matrices. We note that condition (5) is scalable, so the restriction to unit vectors is not essential.
An matrix satisfies a statistical incoherence condition (is -SINC) if
This condition is discussed in [29, 47], and more explicitly in . Following , it appears in the proofs of sparse recovery in  and below in this paper. A somewhat similar average coherence condition was also introduced in [3, 4]. The reason that (6) is less restrictive than the coherence property is as follows. Collections of unit vectors with small coherence (large separation) cannot be too large so as not to contradict universal bounds on packings of At the same time, for the norm to be large it is necessary that a given column is close to the majority of the vectors from the set , which is easier to rule out.
Nevertheless, the above relaxed conditions are still restrictive enough to rule out many deterministic matrices: the problem is that for almost all supports we require that be small for all We observe that this condition can be further relaxed. Namely, let
be the set of all values taken by the coherence parameter. Let us introduce the following definition.
An matrix is said to satisfy a weak statistical incoherence condition (to be a -WSINC) if
where is a positive increasing function of and
We note that this definition is informative if otherwise, replacing it with 1 we get back the SINC condition. Below we use This definition takes account of the distribution of values of the quantity for different choices of the support and a column outside it. For binary dictionaries, the WSINC property relies on a distribution of sums of Hamming distances between a column and a collection of columns, taken with weights that decrease as the sum increases.
We say that a signal is drawn from a generic random signal model if
1) The locations of the coordinates of with largest magnitudes are chosen among all -subsets with a uniform distribution;
2) Conditional on , the signs of the coordinates are i.i.d. uniform Bernoulli random variables taking values in the set .
Using previous defined notation, the probability induced by the generic model can be decomposed as .
1.2. Contributions of this paper
Our results are as follows. First, we show that a combination of the StRIP and SINC conditions suffices for stable recovery of sparse signals. In their large part, these results are due to . We incorporate some additional elements such as stability analysis of Basis Pursuit based on these assumptions and give the explicit values of the constants involved in the assumptions. We also show that the WSINC condition together with StRIP is sufficient for bounding the off-support error of Basis Pursuit.
where and is a constant; see , Theorem B. For this condition to be applicable, one needs that For sampling matrices that satisfy this condition, we obtain a near-optimal relation between the parameters. Some examples of this kind are given below in Sect. 5. As one of our main results, we extend the region of parameters that suffice for -StRIP. Namely, in Theorem 4.7 we prove that it is enough to have the relation . This improvement comes at the expense of an additional requirement on (or a similar inequality for ), but this is easily satisfied in a large class of examples, discussed below in the paper. These examples in conjunction with Theorem 4.1 and the results in Section 2 establish provable error guarantees for some new classes of sampling matrices.
We note a group of papers by Bajwa and Calderbank [3, 4, 13] which is centered around the analysis of a threshold decoding procedure (OST) defined in . The sufficient conditions in these works are formulated in terms of and maximum average coherence Reliance on two coherence parameters of for establishing sufficient conditions for error estimates in  is a shared feature of these papers and our research. At the same time, the OST procedure relies on additional assumptions such as minimum-to-average ratio of signal components bounded away from zero (in experiments, OST is efficient for uniform-looking signals, and is less so for sparse signals with occasional small components). Some other similar assumptions are required for the proofs of the noisy version of OST .
We note that there is a number of other studies that establish sufficient conditions for sampling matrices to provide bounded-error approximations in sparse recovery procedures, e.g., [16, 34, 35]. At the same time, these conditions are formulated in terms different from our assumptions, so no immediate comparison can be made with our results.
As a side result, we also calculate the parameters for the StRIP and SINC conditions that suffice to derive an error estimate for sparse recovery using Lasso. This result is implicit in the work of Candés and Plan , which also uses the SINC property of sampling matrices. The condition on sparsity for Lasso is in the form so if this yields This range of parameters exceeds the range in which Basis Pursuit is shown to have good error guarantees, even with the improvement obtained in our paper. At the same time, both  and our calculations find error estimates in the form of bounds on rather than i.e., on the compressed version of the recovered signal.
In the final section of the paper we collect examples of incoherent dictionaries that satisfy our sufficient conditions for approximate recovery using Basis Pursuit. Two new examples with nearly optimal parameters that emerge are the Delsarte-Goethals dictionaries  and deterministic sub-Fourier dictionaries . For instance, in the Delsarte-Goethals case we obtain the sketch dimension on the order of which is near-optimal, and is in line with the comments made above.
We also show that the restricted independence property of the dictionary suffices to establish the StRIP condition. Using sets of binary vectors known as orthogonal arrays, we find -StRIP dictionaries with At the same time, we are not able to show that restricted independence gives rise to the SINC property with good parameter estimates, so this result has no consequences for linear programming decoders.
Acknowledgment: We are grateful to Waheed Bajwa for useful feedback on an early version of this work.
2. Statistical Incoherence Properties and Basis Pursuit
In this section we prove approximation error bounds for recovery by Basis Pursuit from linear sketches obtained using deterministic matrices with the StRIP and SINC properties.
2.1. StRIP Matrices with incoherence property
It was proved in  that random sparse signals sampled using matrices with the StRIP property can be recovered with high probability from low-dimensional sketches using linear programming. In this section we prove a similar result that in addition incorporates stability analysis.
Suppose that is a generic random signal from the model Let and let be the approximation of by the Basis Pursuit algorithm. Let be the set of largest coordinates of . If
then with probability at least
This theorem implies that if the signal itself is -sparse then the basis pursuit algorithm will recover it exactly. Otherwise, its output will be a tight sparse approximation of .
Theorem 2.1 will follow from the next three lemmas. Some of the ideas involved in their proofs are close to the techniques used in . Let be the error in recovery of basis pursuit. In the following refers to the support of the largest coordinates of
Let Suppose that and
Clearly, so and
Next we show that the error outside cannot be large. Below is a -vector of signs of the argument vector
Suppose that there exists a vector such that
is contained in the row space of , say
By (2) we have
Here we have used the inequality valid for any two vectors and the triangle inequality. From this we obtain
Further, using the properties of we have
The statement of the lemma is now evident.
Now we prove that such a vector as defined in the last lemma indeed exists.
From the definition of it is clear that it belongs to the row-space of and We have where
We will show that for all with probability
Since the coordinates of are i.i.d. uniform random variables taking values in the set , we can use Hoeffding’s inequality to claim that
On the other hand, for all
Using the union bound, we now obtain the following relation:
Hence for all with probability at least .
Now we are ready to prove Theorem 2.1.
Proof of Theorem 2.1.
The matrix is -SRIP. Hence, with probability at least . At the same time, from the SINC assumption we have, with probability at least over the choice of ,
for all Thus, will have these two properties with probability at least . Then from Lemma 2.2 we obtain that
with probability . This completes the proof.
2.2. StRIP Matrices with weak incoherence property
In this section we establish a recovery guarantee of Basis Pursuit under the weak SINC condition defined earlier in the paper.
Suppose that the sampling matrix is -StRIP and -WSINC, where and Suppose that the signal is chosen from the generic random signal model and let be the approximation of found by Basis Pursuit. Then with probability at least we have
If is -sparse and satisfies the condition , then this theorem asserts that Basis Pursuit will find the support of . If in addition is the only -sparse solution to then we have . Note that the WSINC property is not sufficient for the error guarantee. However, once the corrected support is detected, the signal can be found by solving the overcomplete system .
Suppose that the sampling matrix satisfies the conditions of Theorem 2.5. For any and define Let
As in the proof of Lemma 2.4, we define the vector
and let be the th coordinate of the vector From now on we write simply omitting the dependence on and . Let then the StRIP property of implies that
By definition, for any
Now we split the target probability into three parts:
where is the set of supports appearing in the definition of the WSINC property. If i.e., it supports both StRIP and SINC properties, then (11) implies that so the first term on the right-hand side equals 0. The third term refers to supports with no SINC property, whose total probability is Estimating the second term by the Markov inequality, we have
where denotes the indicator random variable. We have
Proof of Theorem 2.5: Define the set by
Recall that Theorem 2.5 is stated with respect to the random signal . Therefore, let us estimate the probability
This implies that with probability the signal chosen from the generic random signal model satisfies the conditions of Lemma 2.3, i.e.,
This completes the proof.
3. Incoherence Properties and Lasso
In this section we prove that sparse signals can be approximately recovered from low-dimensional observations using Lasso if the sampling matrices have statistical incoherence properties. The result is a modification of the methods developed in [15, 49] in that we prove that the conditions used there to bound the error of the Lasso estimate hold with high probability if is has both StRIP and SINC properties. The precise claim is given in the following statement.
Let be a random -sparse signal whose support satisfies the two properties of the generic random signal model Denote by its estimate from via Lasso (3), where is a i.i.d. Gaussian vector with zero mean and variance and where Suppose that , where is a positive constant, and that the matrix satisfies the following two properties:
Then we have
with probability at least , where is an absolute constant and
The following theorem is implicit in , see Theorem 1.2 and Sect 3.2 in that paper.
(Candès and Plan) Suppose that is a -sparse signal drawn from the model where
where is a constant. Let be the support of and suppose the following three conditions are satisfied:
where is an absolute constant.
Our aim will be to prove that conditions (1)-(3) of this theorem hold with large probability under the assumptions of Theorem 3.1.
First, it is clear that with probability at least This follows simply because is an independent Gaussian vector, and has been discussed in  (this is also the reason for selecting the particular value of ). The main part of the argument is contained in the following lemma whose proof uses some ideas of .
Suppose that and that for all
Then Condition (3) of Theorem 3.2 holds with probability at least for .
Let Define and where
Let and We will show that with high probability and