On the Lagrangian Biduality of Sparsity Minimization Problems
Recent results in Compressive Sensing have shown that, under certain conditions, the solution to an underdetermined system of linear equations with sparsity-based regularization can be accurately recovered by solving convex relaxations of the original problem. In this work, we present a novel primal-dual analysis on a class of sparsity minimization problems. We show that the Lagrangian bidual (i.e., the Lagrangian dual of the Lagrangian dual) of the sparsity minimization problems can be used to derive interesting convex relaxations: the bidual of the -minimization problem is the -minimization problem; and the bidual of the -minimization problem for enforcing group sparsity on structured data is the -minimization problem. The analysis provides a means to compute per-instance non-trivial lower bounds on the (group) sparsity of the desired solutions. In a real-world application, the bidual relaxation improves the performance of a sparsity-based classification framework applied to robust face recognition.
The last decade has seen a renewed interest in the problem of solving an underdetermined system of equations , , where , by regularizing its solution to be sparse, i.e., having very few non-zero entries. Specifically, if one aims to find with the least number of nonzero entries that solves the linear system, the problem is known as -minimization:
The problem is intended to seek entry-wise sparsity in and is known to be NP-hard in general. In Compressive Sensing (CS) literature, it has been shown that the solution to (1) often can be obtained by solving a more tractable linear program, namely, -minimization [8, 4]:
This unconventional equivalence relation between and and the more recent numerical solutions [3, 16] to efficiently recover high-dimensional sparse signal have been a very competitive research area in CS. Its broad applications have included sparse error correction , compressive imaging , image denoising and restoration [11, 17], and face recognition [21, 13], to name a few.
In addition to enforcing entry-wise sparsity in a linear system of equations, the notion of group sparsity has attracted increasing attention in recent years [18, 12, 13]. In this case, one assumes that the matrix has some underlying structure, and can be grouped into blocks: , where and . Accordingly, the vector is split into several blocks as , where . In this case, it is of interest to estimate with the least number of blocks containing non-zero entries. The group sparsity minimization problem is posed as
where is the indicator function. Since the expression can be written as , it is also denoted as , the -norm of .
Enforcing group sparsity exploits the problem’s underlying structure and can improve the solution’s interpretability. For example, in a sparsity-based classification (SBC) framework applied to face recognition, the columns of are vectorized training images of human faces that can be naturally grouped into blocks corresponding to different subject classes, is a vectorized query image, and the entries in represent the coefficients of linear combination of all the training images for reconstructing . Group sparsity lends itself naturally to this problem since it is desirable to use images of the smallest number of subject classes to reconstruct and subsequently classify a query image.
Furthermore, the problem of robust face recognition has considered an interesting modification known as the cross-and-bouquet (CAB) model: , where represents possible sparse error corruption on the observation . It can be argued that the CAB model can be solved as a group sparsity problem in (3), where the coefficients of would be the group. However, this problem has a trivial solution for and , which would have the smallest possible group sparsity. Hence, it is necessary to further regularize the entry-wise sparsity in .
To this effect, one considers a mixture of the previous two cases, where one aims to enforce entry-wise sparsity as well as group sparsity such that has very few number of non-zero blocks and the reconstruction error is also sparse. The mixed sparsity minimization problem can be posed as
where controls the tradeoff between the entry-wise sparsity and group sparsity.
Due to the use of the counting norm, the optimization problems in (3) and (4) are also NP-hard in general. Hence, several recent works have focused on developing tractable convex relaxations for these problems. In the case of group sparsity, the relaxation involves replacing the -norm with the -norm, where . These relaxations are also used for the mixed sparsity case .
In this work, we are interested in deriving and analyzing convex relaxations for general sparsity minimization problems. In the entry-wise case, the main theoretical understanding of the link between the original NP-hard problem in (1) and its convex relaxation has been given by the simple fact that the -norm is a convex surrogate of the -norm. However, in the group sparsity case, a similar relaxation produces a family of convex surrogates, i.e., , whose value depends on . This raises the question whether there is a preferable value of for the relaxation of the group sparsity minimization problem? In fact, we consider the following more important question:
Is there a unified framework for deriving convex relaxations of general sparsity recovery problems?
1.1 Paper contributions
We present a new optimization-theoretic framework based on Lagrangian duality for deriving convex relaxations of sparsity minimization problems. Specifically, we introduce a new class of equivalent optimization problems for , and , and derive the Lagrangian duals of the original NP-hard problems. We then consider the Lagrangian dual of the Lagrangian dual to get a new optimization problem that we term as the Lagrangian bidual of the primal problem. We show that the Lagrangian biduals are convex relaxations of the original sparsity minimization problems. Importantly, we show that the Lagrangian biduals for the and problems correspond to minimizing the -norm and the -norm, respectively.
Since the Lagrangian duals for , and are linear programs, there is no duality gap between the Lagrangian duals and the corresponding Lagrangian biduals. Therefore, the bidual based convex relaxations can be interpreted as maximizing the Lagrangian duals of the original sparsity minimization problems. This provides new interpretations for the relaxations of sparsity minimization problems. Moreover, since the Lagrangian dual of a minimization problem provides a lower bound for the optimal value of the primal problem, we show that the optimal objective value of the convex relaxation provides a non-trivial lower bound on the sparsity of the true solution to the primal problem.
2 Lagrangian biduals for sparsity minimization problems
In what follows, we will derive the Lagrangian bidual for the mixed sparsity minimization problem, which generalizes the entry-wise sparsity and group sparsity cases (also see Section 3). Specifically, we will derive the Lagrangian bidual for the following optimization problem:
where and . Given any unique, finite solution to (5), there exists a constant such that the absolute values of the entries of are less than , namely, . Note that if (5) does not have a unique solution, it might not be possible to choose a finite-valued that upper bounds all the solutions. In this case, a finite-valued may be viewed as a regularization term for the desired solution. To this effect, we consider the following modified version of (5) where we introduce the box constraint that :
Primal problem. We will now frame an equivalent optimization problem for (6), for which we introduce some new notation. Let be an entry-based sparsity indicator for , namely, if and otherwise. We also introduce a group-based sparsity indicator vector , whose entry denotes whether the block contains non-zero entries or not, namely, if and otherwise. To express this constraint, we introduce a matrix , such that if the entry of belongs to the block and otherwise. Finally, we denote the positive component and negative component of as and , respectively, such that .
Given these definitions, we see that (6) can be reformulated as
where and .
Constraints (a)–(d) are used to enforce the aforementioned conditions on the values of the solution. While constraint (e) enforces the condition that the original system of linear equations is satisfied, the constraints (f) and (g) ensure that the group sparsity indicator and the entry-wise sparsity indicator are consistent with the entries of .
Lagrangial dual. The Lagrangian function for (7) is given as
where , , , and . In order to obtain the Lagrangian dual function, we need to minimize with respect to , , and . Notice that if the coefficients of and , i.e., and are non-zero, the minimization of with respect to and is unbounded below. To this effect, the constraints that these coefficients are equal to form constraints on the dual variables. Next, consider the minimization of with respect to . Since each entry only takes values or , its optimal value that minimizes is given as
A similar expression can be computed for the minimization with respect to . As a consequence, the Lagrangian dual problem can be derived as
This can be further simplified by rewriting it as the following linear program:
Notice that we have made two changes in going from (10) to (11). First, we have replaced constraints (b) and (c) in (10) with the constraint (g) in (11) and eliminated and from (11). Second, we have introduced variables and to encode the “min” operator in the objective function of (10).
Notice that in going from (7) to (12), the discrete valued variables and have been relaxed to take real values between and . Given that and noting that can be represented as , we can conclude from constraint (g) in (12) that the solution satisfies . Moreover, given that and are relaxed to take real values, we see that the optimal values for and are and , respectively. Hence, we can eliminate constraints (f) and (g) by replacing and by these optimal values. It can then be verified that solving (12) is equivalent to solving the problem:
This is the Lagrangian bidual for (7).
3 Theoretical results from the biduality framework
In this section, we first describe some properties of the biduality framework in general. We will then focus on some important results for the special cases of entry-wise sparsity and group sparsity.
Since there is no duality gap between a linear program and its Lagrangian dual , the optimal values of the Lagrangian dual in (11) and the Lagrangian bidual in (13) are the same. Moreover, we know that the optimal value of a primal minimization problem is always bounded below by the optimal value of its Lagrangian dual . We hence have the required result. ∎
Since the original primal problem in (7) is NP-hard, we note that the duality gap between the primal and its dual in (11) is non-zero in general. Moreover, we notice that as we increase (i.e., a more conservative estimate), the optimal value of the primal is unchanged, but the optimal value of the bidual in (13) decreases. Hence, the duality gap increases as increases.
in (6) should preferably be equal to , which may not be possible to estimate accurately in practice. Therefore, it is of interest to analyze the effect of taking a very conservative estimate of , i.e., choosing a large value for . In what follows, we show that taking a conservative estimate of is equivalent to dropping the box constraint in the bidual.
For this purpose, consider the following modification of the bidual:
where we have essentially dropped the box constraint (b) in (13). It is easy to verify that , we have that . Therefore, we see that taking a conservative value of is equivalent to solving the modified bidual in (14).
3.1 Results for entry-wise sparsity minimization
Notice that by substituting and , the optimization problem in (5) reduces to the entry-wise sparsity minimization problem in (1). Hence, the Lagrangian bidual to the -regularized entry-wise sparsity problem is:
More importantly, we can also conclude from (14) that solving the Lagrangian bidual to the entry-wise sparsity problem with a conservative estimate of is equivalent to solving the problem:
which is precisely the well-known -norm relaxation for . Our framework therefore provides a new interpretation for this relaxation:
We further note that we can now use the solution of (15) to derive a non-trivial lower bound for the primal objective function which is precisely the sparsity of the desired solution. More specifically, we can use Theorem 1 to conclude the following result:
Let be the solution to (1). We have that , the sparsity of , i.e., is bounded below by .
Due to the non-zero duality gap in the primal entry-wise sparsity minimization problem, the above lower bound provided by Corollary 1 is not tight in general.
3.2 Results for group sparsity minimization
As in the case of entry-wise sparsity above, solving the bidual to the group sparsity problem with a conservative estimate of is equivalent to solving:
which is the convex -norm relaxation of the -min problem (3). In other words, the biduality framework selects the -norm out of the entire family of -norms as the convex surrogate of the -norm.
Finally, we use Theorem 1 to show that the solution obtained by minimizing the -norm provides a lower bound for the group sparsity.
Let be the solution to (3). For any , the group sparsity of , i.e., , is bounded below by .
The -norm seems to be an interesting choice for computing the lower bound of the group sparsity, as compared to other -norms for finite . For example, consider the case when , where the -norm is equivalent to the -norm. Assume that consists of a single block with several columns so that the maximum number of non-zero blocks is . Denote the solution to the -minimization problem as . It is possible to construct examples (also see Figure 1) where . Hence, it is unclear in general if the solutions obtained by minimizing -norms for finite-valued can help provide lower bounds for the group sparsity.
We now present experiments to evaluate the bidual framework for minimizing entry-wise sparsity and mixed sparsity. We present experiments on synthetic data to show that our framework can be used to compute non-trivial lower bounds for the entry-wise sparsity minimization problem. We then consider the face recognition problem where we compare the performance of the bidual-based -norm relaxation with that of the -norm relaxation for mixed sparsity minimization.
We use boxplots to provide a concise representation of our results’ statistics. The top and bottom edge of a boxplot for a set of values indicates the maximum and minimum of the values. The bottom and top extents of the box indicate the and percentile mark. The red mark in the box indicates the median and the red crosses outside the boxes indicate potential outliers.
Entry-wise sparsity. We now explore the practical implications of Corollary 1 through synthetic experiments. We randomly generate entries of and from a Gaussian distribution with unit variance. The sparsity of is varied from to in steps of . We solve (15) with using and , where . We use Corollary 1 to compute lower bounds on the true sparsity, i.e., . We repeat this experiment times for each sparsity level and Figure 1 shows the boxplots for the bounds computed from these experiments.
We first analyze the lower bounds computed when , in Figure 1(a). As explained in Section 3.1, the bounds are not expected to be tight due to the duality gap. Notice that for extremely sparse solutions, the maximum of the computed bounds is close to the true sparsity but this diverges as the sparsity of reduces. The median value of the bounds is much looser and we see that the median also diverges as the sparsity of reduces. Furthermore, the computed lower bounds seem to grow linearly as a function of the true sparsity. Similar trends are observed for and in Figures 1(b) and 1(c), respectively. As expected from the discussion in Section 3.1, the bounds become very loose as increases.
In theory, we would like to have per-instance certificates-of-optimality of the computed solution, where the lower bound is equal to the true sparsity . Nonetheless, we note that this ability to compute a per-instance non-trivial lower bound on the sparsity of the desired solution is an important step forward with respect to the previous approaches that require pre-computing optimality conditions for equivalence of solutions to the -norm and -norm minimization problems.
We have performed a similar experiment for the group sparsity case, and observed that the bidual framework is able to provide non-trivial lower bounds for the group sparsity also.
Mixed sparsity. We now evaluate the results of mixed sparsity minimization for the sparsity-based face recognition problem, where the columns of represent training images from the face classes: and represents a query image. We assume that a subset of pixel values in the query image may be corrupted or disguised. Hence, the error in the image space is modeled by a sparse error term : , where is the uncorrupted image. A linear representation of the query image forms the following linear system of equations:
where is the identity matrix. The goal of sparsity-based classification (SBC) is to minimize the group sparsity in and the sparsity of such that the dominant non-zero coefficients in reveal the membership of the ground-truth observation [21, 13]. In our experiments, we solve for and by solving the following optimization problem:
Notice that for , this reduces to solving a special case of the problem in (14), i.e., the bidual relaxation of the mixed sparsity problem with a conservative estimate of . In our experiments, we set and compare the solutions to (20) obtained using and .
We evaluate the algorithms on a subset of the AR dataset  which has manually aligned frontal face images of size for 50 male and 50 female subjects, i.e., and . Each individual contributes 7 un-occluded training images, 7 un-occluded testing images and 12 occluded testing images. Hence, we have 700 training images and 1900 testing images. To compute the number of non-zero blocks in the coefficient estimated for a testing image, we find the number of blocks whose energy is greater than a specified threshold.
The results of our experiments are presented in Figure 2. The solution obtained with gives better group sparsity of . However, a sparser error is estimated with . The number of non-zero entities in a solution to (20), i.e., the number of non-zero blocks plus the number of non-zero error entries, is lower for the solution obtained using rather than that obtained using . However, the primal mixed-sparsity objective value (see (4)) is lower for the solution obtained using .
We now compare the classification results obtained with the solutions computed in our experiments. For classification, we consider the non-zero blocks in and then assign the query image to the block, i.e., subject class, for which it gives the least residual . The results are presented in Table 1. Notice that the classification results obtained with (the bidual relaxation) are better than those obtained using . Since the classification of un-occluded images is already very good using , classification with gives only a minor improvement in this case. However, a more tangible improvement is noticed in the classification of the occluded images. Therefore the classification with is in general better than that obtained with , which is considered the state-of-the-art for sparsity-based classification .
|(correct results)||(correct results)||(correct results)||(correct results)|
We have presented a novel analysis of several sparsity minimization problems which allows us to interpret several convex relaxations of the original NP-hard primal problems as being equivalent to maximizing their Lagrangian duals. The pivotal point of this analysis is the formulation of mixed-integer programs which are equivalent to the original primal problems. While we have derived the biduals for only a few sparsity minimization problems, the same techniques can also be used to easily derive convex relaxations for other sparsity minimization problems .
An interesting result of our biduality framework is the ability to compute a per-instance certificate of optimality by providing a lower bound for the primal objective function. This is in contrast to most previous research which aims to characterize either the subset of solutions or the set of conditions for perfect sparsity recovery using the convex relaxations [6, 5, 8, 9, 10, 14, 15, 20]. In most cases, the conditions are either weak or hard to verify. More importantly, these conditions needed to be pre-computed as opposed to verifying the correctness of a solution at run-time. In lieu of this, we hope that our proposed framework will prove an important step towards per-instance verification of the solutions. Specifically, it is of interest in the future to explore tighter relaxations for the verification of the solutions.
This research was supported in part by ARO MURI W911NF-06-1-0076, ARL MAST-CTA W911NF-08-2-0004, NSF CNS-0931805, NSF CNS-0941463 and NSF grant 0834470. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute for Government purposes notwithstanding any copyright notation herein.
- A. M. Martinez, and, R. Benavente, “The AR face database.” CVC Technical Report 24, 1998.
- S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
- A. Bruckstein, D. Donoho, and M. Elad. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review, 51(1):34–81, 2009.
- E. Candès. Compressive sampling. In Proceedings of International Congress of Mathematics, 2006.
- E. Candés. The restricted isometry property and its implications for compressed sensing. C. R. Acad. Sci., Paris, Series I, 346: 589–592, 2008.
- E. Candès and T. Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51(12):4203–4215, 2005.
- V. Cevher, M. F. Duarte, C. Hegde, R. G.. Baraniuk. Sparse signal recovery using Markov random fields. In NIPS 257–264, 2008.
- D. Donoho and M. Elad. Optimally sparse representation in general (nonorthogonal) dictionaries via minimization. PNAS, 100(5):2197–2202, 2003.
- D. Donoho. For most large underdetermined systems of linear equations, the minimal -norm near-solution approximates the sparsest near-solution. Communications on Pure and Applied Mathematics. 2006.
- D. Donoho, and M. Elad, and V. N. Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. on Information Theory, 52(1): 6–18, 2006.
- M. Protter, M. Elad. Image sequence denoising via sparse and redundant representations. IEEE Transactions on Image Processing 18(1): 27–35, 2009.
- Y. Eldar and M. Mishali. Robust recovery of signals from a structured union of subspaces. IEEE Transactions on Information Theory, 55(11):5302–5316, 2009.
- E. Elhamifar and R. Vidal. Robust classification using structured sparse representation. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.
- A. Fletcher, S. Rangan, and V. Goyal. Necessary and sufficient conditions on sparsity pattern recovery. IEEE Transactions on Information Theory, 55(12): 5758-5772 (2009)
- A. Iouditski, F. K. Karzan, and A. Nemirovski. Verifiable conditions of -recovery of sparse signals with sign restrictions. ArXiv e-prints, 2009.
- I. Loris. On the performance of algorithms for the minimization of -penalized functionals. Inverse Problems, 25:1–16, 2009.
- J. Mairal, M. Elad and G. Sapiro. Sparse representation for color image restoration. IEEE Transactions on Image Processing 17(1): 53-69, 2008.
- M. Stojnic, F. Parvaresh, B. Hassibi. On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Transactions on Signal Processing 57(8): 3075–3085, 2009.
- J. Tropp. Greed is Good: Algorithmic results for sparse approximation. IEEE Transactions on Information Theory 50(10): 2231–2242, 2004.
- G. Reeves and M. Gastpar. Efficient sparsity pattern recovery. In Proc. 30th Symp. on Information Theory Benelux, 2009.
- J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2):210–227, 2009.
- J. Wright and Y. Ma. Dense Error Correction via -Minimization. IEEE Transactions on Information Theory, 56(7), 2010.
- M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly and R. Baraniuk. Compressive imaging for video representation and coding. In Picture Coding Symposium, 2006.