On the Bit Complexity of Sum-of-Squares Proofs
It has often been claimed in recent papers that one can find a degree Sum-of-Squares proof if one exists via the Ellipsoid algorithm. In [Oâ17], Ryan O’Donnell notes this widely quoted claim is not necessarily true. He presents an example of a polynomial system with bounded coefficients that admits low-degree proofs of non-negativity, but these proofs necessarily involve numbers with an exponential number of bits, causing the Ellipsoid algorithm to take exponential time. In this paper we obtain both positive and negative results on the bit complexity of SoS proofs.
First, we propose a sufficient condition on a polynomial system that implies a bound on the coefficients in an SoS proof. We demonstrate that this sufficient condition is applicable for common use-cases of the SoS algorithm, such as Max-CSP, Balanced Separator, Max-Clique, Max-Bisection, and Unit-Vector constraints.
On the negative side, O’Donnell asked whether every polynomial system containing Boolean constraints admits proofs of polynomial bit complexity. We answer this question in the negative, giving a counterexample system and non-negative polynomial which has degree two SoS proofs, but no SoS proof with small coefficients until degree .
The Sum of squares (SoS) proof system is a versatile and powerful approach to certifying polynomial inequalities. SoS certificates can be shown to underly a vast number of algorithms in combinatorial optimization. On the one hand, SoS certificates hold the promise of yielding algorithms that possibly refute the notorious unique games conjecture [BBH12, BRS11, GS11]. On the other hand, a flurry of recent works have applied SoS proofs to develop algorithms for problems ranging from constraint satisfaction problems to tensor problems.
To illustrate sum of squares certificates, let us consider the example of the Balanced Separator problem. Here we are given a graph and the goal is to find a balanced cut with the minimum number of crossing edges. Like many problems in combinatorial optimization, it can be reformulated as a low-degree polynomial optimization problem. Specifically if we associate variables for the vertices of the graph then we can rewrite the Balanced Separator problem as follows,
Here the constraint ensures while the inequalities enforce the condition that the cut is balanced. More generally, a low-degree polynomial optimization is of the form
An SoS certificate of a lower bound is given by a polynomial identity of the form
Notice that for all satisfying the equalities and the inequalities , the right hand side of the above identity is manifestly non-negative, thereby certifying that . The degree of the SoS certificate is the maximum degree of the polynomials involved, i.e., .
The main appeal of SoS certificates for polynomial optimization is that the existence of a degree SoS certificate can be formulated as the feasibility of a semidefinite program (SDP). This is the degree SoS relaxation first introduced by Shor [Sho87], and expanded upon by later works of Nesterov [Nes00], Grigoriev and Vorobjov [GV01], Lasserre [Las00, Las01] and Parrilo [Par00]. (see, e.g., [Lau09, BS14] for many more details).
The degree SoS SDP has variables, and if the coefficients of and are reasonably bounded (smaller than ), the resulting SDP has a compact description of size . From this, several works including those by the authors, asserted that the resulting feasibility SDP can be solved in time using the Ellipsoid algorithm.
In a recent work, O’Donnell [Oâ17] observed that this often repeated claim is far from true. Specifically, O’Donnell exhibited systems of polynomial inequalities with bounded coefficients such that only degree SoS certificates of non-negativity involve coefficients that are doubly exponential in size. Thus all SoS certificates need an exponential number of bits to represent and consequently, the ellipsoid algorithm will incur an exponential running time.
As pointed out by O’Donnell, the issue at hand here is not just that of additive error in the solution, i.e., the difference between testing feasibility and near-feasibility. Indeed, semidefinite programming via the ellipsoid algorithm can only test feasibility up to a very small additive error. However, in a majority of applications of SoS SDP relaxations in combinatorial optimization, the variables in the underlying polynomial system are explictly bounded (also known as Archimedian). Specifically, these include constraints such as , which yield explicit bounds on the values of the variables. In these settings, if there is an approximate SoS certificate for , then there exists a proper SoS certificate for a slightly weaker lower bound . Therefore, additive error incurred in semidefinite programming can often be traded off for a slightly weaker objective value. The issue highlighted by O’Donnell is far more serious in that the coefficients of the SoS certificate are too large – thereby directly affecting the runtime of the ellipsoid algorithm.
On a positive note, O’Donnell shows that a polynomial system whose only constraints are the Boolean constraints always admit SoS certificates with polynomial bit complexity. He proceeds to ask whether all polynomial systems that include boolean constraints, potentially among others, always admit bounded SoS certificates.
1.1 Our Results
In this work, we further explore the issue of bit complexity of SoS proofs, and obtain both positive and negative results.
First, we present an easily verifiable and broadly applicable set of sufficient conditions under which a polynomial optimization problem has small SoS certificates. Roughly speaking, we show that polynomial systems with rich sets of solutions have bounded SoS certificates of non-negativity. Consider a system consisting of polynomial equalities and inequalities . Our approach consists of looking for assignments satisfying three criteria (see Definition LABEL:*def:nice and Theorem LABEL:*thm:main for formal statements).
The assignments robustly satisfy the inequalities in .
The polynomial calculus proof system is both complete and efficient over . In other words, all degree polynomial identities over can be derived using a degree polynomial derivation from the equalities .
The assignments are spectrally rich in that smallest non-zero eigenvalue of their covariance matrix is at least .
Then if has a degree proof of non-negativity from and , it also has a degree proof of non-negativity with coefficients bounded by .
We demonstrate the broad applicability of the above set of sufficient conditions by using them to show upper bounds on bit complexity for Max-CSP, Max-Clique, Matching, Balanced Separator, Max-Bisection, and optimization over the unit sphere. In each case, the above sufficient conditions can be verified easily.
The above set of sufficient conditions are widely applicable in combinatorial optimization, wherein the polynomial system is typically a relaxation of a well-known set of integer solutions. In such a setup with integer solutions, we observe in Section LABEL:*sec:nicespaces that spectral richness is an immediate consequence of the discrete nature of the set of solutions. Therefore, in all these setups, the only non-trivial thing to verify is the efficiency of the polynomial calculus proof system.
The work of O’Donnell [Oâ17] exhibited a polynomial system with bounded coefficients which admitted degree SoS certificate, whose coefficients were necessarily doubly-exponential. However, the variables in this polynomial system were not all boolean, i.e. did not have the constraint. In fact, O’Donnell asked whether every polynomial system with boolean constraints admits a small SoS proof. Moreover, the polynomial system in [Oâ17] admits a degree SoS certificate with small bit complexity. This opens up the possibility that one can effectively reduce the bit-complexity by raising the degree of the proof. For instance, if a system admits a degree SoS certificate then does it always admit a degree SoS certificate with small bit complexity (even under boolean constraints)? Unfortunately, we refute both of the above possibilities by exhibiting a counterexample. Formally, we show the following:
There exists a system of quadratic equations on variables such that
The system includes the equation for each .
There exists a polynomial with a degree SoS certificate of non-negativity, albeit with doubly exponentially large coefficients.
No SoS certificate of degree has coefficients smaller than .
For a set of real polynomials , we denote their generated ideal in by or . We will be working with systems of polynomial constraints, and we will use the to denote the equality constraints, and to denote the inequality constraints, i.e. and for and . We will usually use for the set of points satisfying these constraints. We use for the set of polynomials of degree at most , and we write for the vector of polynomials whose entries are the elements of the usual monomial basis of . Similarly, we use for the vector of reals whose entries are the entries of evaluated at . We usually omit the dependencies on as it is clear from context.
2.1 Polynomial Proofs
Let be a set of polynomials, and let . We define a proof of membership in as follows:
We say that has a derivation from if there is a polynomial identity of the form
We say that the proof has degree if .
The above proof system is useful for proving when polynomials are zero on , but often we want to prove that they are positive. To that end, let and be two sets of polynomials, and let . We define a proof of non-negativity as follows:
We say that has a Sum-of-Squares proof of non-negativity from and if there is a polynomial identity of the form
We say the proof has degree if .
The idea behind this terminology is that if such a proof exists, then must be non-negative on since the first two terms are non-negative, and the last term is zero. We will be concerned with not just the degree of these proofs, but also their bit complexity. To this end, we define the following norms on polynomials and proofs: For , we write for the absolute value of the maximum coefficient of in the standard monomial basis, and for any collection of polynomials , we write . We will also require a bound on . Throughout this paper we will assume that the solutions are explicitly bounded by .
2.2 Rich Solution Spaces
In this section we define the conditions we require in order to guarantee that SoS proofs from and have low bit-complexity. For a set of assignments to a polynomial system , define the moment matrix as
where the expectation is over the uniform distribution over . We will omit the subscript and write , if is clear from the context.
With the above definitions,
We say that is -spectrally rich for up to degree if every nonzero eigenvalue of is at least .
We say that is -complete on up to degree if every zero eigenvector of (which can be seen as a degree polynomial ) has a degree derivation from .
We say that is -robust for if .
Spectral richness of the solutions is equivalent to requiring if is small on , then there is a polynomial which agrees with on and that has small coefficients. If satisfies all three conditions then we say that is -rich for up to degree . If , , and we simply say is rich for . We choose these bounds because Theorem LABEL:*thm:main will imply that any constraints with a rich solution space has proofs of non-negativity that can be taken to have polynomial bit complexity. Before we get into the proof of the main theorem, we exhibit polynomial systems that admit rich solutions.
3 Examples with Rich Solution Spaces
In this section we present examples of polynomial systems that admit rich solution spaces. First, we consider the case . In this case, the spectral richness is a consequence of the following easy observation.
Let be an integer matrix with for all . The smallest non-zero eigenvalue of is at least .
Let be a full-rank principal minor of and w.l.o.g. let it be at the upper-left block of . We claim the least eigenvalue of lower bounds the least nonzero eigenvalue of . Since is symmetric, there must be a such that
Let , be the least eigenvalue of , and be a vector perpendicular to the zero eigenspace of . Then we have , but is also perpendicular to the zero eigenspace of . Now has the same nonzero eigenvalues as , and thus , and so every nonzero eigenvalue of is at least . Now is a full-rank bounded integer matrix with dimension at most . The magnitude of its determinant is at least and all eigenvalues are at most . Therefore, its least eigenvalue must be at least in magnitude. ∎
Let and be such that . Then is -spectrally rich with .
Recall , and note that is an integer matrix with entries at most . The proof follows by applying Lemma LABEL:*lem:integer. ∎
To prove completeness, we typically want to show two things. First, that every degree polynomial in has a degree at most derivation. Second, that there are no polynomials outside that are zero on . This second condition can be thought of as saying that the set of equations is somehow maximal, i.e., if there are extra polynomial equalities implied by , they should be included in . Here we consider a few examples.
Here . Any polynomial of degree can be multilinearized one monomial at a time. Specifically, we can find degree multilinear such that has a degree derivation from . Furthermore, the multilinear polynomial is zero over if and only if all its coefficients are zero. Thus is -complete up to degree for all .
Here is the set of all cliques in the graph. Suppose is a polynomial that is identically zero over . Without loss of generality, we may assume is multilinear, if otherwise we can multilinearize it using . For a multilinear polynomial , we claim that if then for all cliques , the corresponding coefficient , i.e., all non-zero coefficients of are non-cliques. Suppose not, then let be the smallest clique with . Then, we will have , a contradiction. Since all coefficients of are non-cliques, each monomial in can be eliminated using an appropriate polynomial from .
More generally, the above two cases are special cases of the following general setup: is empty, and is a Gröbner basis. A Gröbner basis for an ideal is a generating set of polynomials that allow a well-defined multivariate polynomial division (see [AL94] for more information). Computing the Gröbner basis is often the first step in practical polynomial equation solvers, and we note the following easy lemma:
If and is a Gröbner basis for , then is -complete up to degree .
If is a Gröbner basis, then every degree polynomial in has a degree derivation via multivariate division. Because , the polynomials that are zero on are exactly the polynomials in . ∎
Balanced Separator: ,
The solution space here is all bit strings with hamming weight between and . Suppose is a polynomial that is zero on . Without loss of generality, we may assume that is multilinear by using the constraints . Suppose is a non-zero multilinear polynomial which is zero on , then its symmetrized version must also be zero on , where acts by permuting the variable names. However, is a univariate polynomial in (modulo the Boolean constraints). This univariate polynomial has zeros, and thus must have degree at least . Since symmetrizing doesn’t change degree, we conclude that also has degree at least . Thus every non-zero multilinear polynomial that is zero on but not in , has degree at least . Therefore the system is -complete up to degree for . The polynomials in can be perturbed by to make them -robust, and thus is rich for .
These constraints are -complete as proven in [BBCH16].
We will prove in Section LABEL:*sec:balance that these constraints are -complete. The proof will be very similar to the one for Matching, due to the similar symmetry of the constraints.
Here . This constraint appears frequently in tensor norm problems as a way to enforce scaling. Since , it is clearly robust. It may be well-known that is -complete, but we could not find a reference so we record it here for completeness. Let be any degree polynomial which is zero on the unit sphere, and define . Clearly is also zero on the unit sphere, with degree . Note that has only terms of even degree. Define a sequence of polynomials as follows. Define to be the part of which has degree strictly less than , and let . Then each is zero on the unit sphere and has no monomials of degree strictly less than . Thus is homogeneous of degree . But then for any unit vector and , and thus must be the zero polynomial. This implies that is a multiple of . The same logic shows that is also a multiple of , and thus so is . Now is principal, so every degree polynomial in it has a degree derivation, so is -complete.
To prove spectral-richness, we note that in [Fol01] the author gives an exact formula for each entry of the matrix for any polynomial . The formulas imply that is an integer matrix with entries (very loosely) bounded by . By Lemma LABEL:*lem:integer, we conclude that is -spectrally rich with .
We collect the examples discussed in this section here:
The following constraints admit rich solutions:
Balanced Separator: , .
While Theorem LABEL:*thm:main allows us to prove that many different systems of polynomial constraints have well-behaved SoS proofs, there are a few areas where it comes up short. Most noticeably, to contain a rich set of solutions the solution space has to be nonempty. This can be a problem when trying to find SoS proofs of infeasibility. For example, one common technique is to introduce lower bounds on an objective function of a maximization problem as constraints and attempt to use SoS to find a refutation, i.e. a proof of non-negativity for the constant polynomial . We are unable to show that these proofs can be taken to have polynomial bit complexity since they have empty solution spaces. As another example, we are unable to use our framework to show that refutations of the knapsack constraints use only polynomially many bits, even though it is clear by simply examining these known refutations that they only involve small coefficients.
4 Rich Solution Spaces Yield Bounded SoS Proofs
In this section we prove our main theorem:
Let and be sets of polynomials with . Assume that the set of solutions is -rich for .
Let be a polynomial nonnegative on , and assume has a degree sum-of-squares proof of nonnegativity
Then has a degree sum-of-squares proof of nonnegativity such that the coefficients of every polynomial appearing in the proof are bounded by . In particular, if is rich then every coefficient can be written down with only bits.
First, we rewrite the proof into a more convenient form before proving bounds on each individual term. Because the elements of are a basis for , every polynomial in the proof can be expressed as , where is a vector of reals:
for PSD matrices , . Next, we average this polynomial identity over all the points :
The LHS is at most , and the RHS is a sum of positive numbers, so the LHS is a bound on each term of the RHS. We would like to say that since is -spectrally rich, the first term is at least . Unfortunately the averaged matrix may have zero eigenvectors, and it is possible that could have very large eigenvalues in these directions. However these eigenvectors must correspond to polynomials that are zero on . Because is complete, these can be absorbed into the final term. More formally, let be the projector onto the zero eigenspace of . Because is complete, for each we have a degree derivation . Then . Thus we can write
Doing the same for the other terms and setting and similarly for , we get a new proof:
Now after averaging over , the zero eigenspace of is contained in the zero eigenspace of . Taken with the -spectral richness, we have
Because each , we get and have entries bounded by .
The only thing left to do is to bound the coefficients , but this is easy because the SoS proof is linear in these coefficients. If we imagine the coefficients of the as variables, then the linear system induced by the polynomial identity
is clearly feasible, and the coefficients of the LHS are bounded by . There are variables, so by Cramer’s rule, the coefficients of the can be taken to be bounded by . as they are considered part of the input, by the explicitly bounded assumption, and . Thus, this bound is at most .
5 Boolean Systems With No Small-Coefficient Proofs
In [Oâ17], the author gives an example of a polynomial system for which degree two SoS proofs can certify non-negativity of a certain poylnomial, but the proofs necessarily involves coefficients of doubly-exponential size. However, there are two weaknesses in his example system. First, it is not a Boolean one, i.e. it contains variables for which the constraint is not present in the constraints. Many practical optimization problems have Boolean constraints, and in [Oâ17], the author hoped that having those constraints might suffice to imply that all proofs could have small bit complexity. Second, while the degree two proofs must have exponential bit complexity, there were degree four proofs of non-negativity with polynomial bit complexity. In this section, we strengthen his counterexample, giving an example of a Boolean system with variables for which there is a polynomial that has a degree two proof of non-negativity, but no proof with polynomial bit complexity until degree .
5.1 A First Example
The original example given in [Oâ17] essentially contains the following system whose repeated squaring is responsible for the blowup of the coefficients in the proofs:
Clearly, the only solution to the system is , and therefore the polynomial must be non-negative over the solution space for any . It is not as obvious whether or not an SoS proof of this non-negativity exists. It turns out that there is a degree two SoS proof as follows:
where the is equality modulo the ideal generated by the constraints. Of course, this proof involves coefficients of doubly-exponential size, but one can prove that they are required. We will take for simplicity. We will define a linear functional satisfying the following:
for any of degree at most
for any and of degree at most
If such a exists, then for any degree SoS proof of non-negativity
apply to both sides. We obtain , where . Because , must contain a coefficient of size at least .
To show that such a exists, we define it as follows. By the constraints, every monomial is equivalent to some power of . For example, . More generally, the constraints imply that . Define by,
One can easily check that this satisfies the above. Note that none of the variables in the above system are boolean, which we achieve in the upcoming section.
5.2 A Boolean System
One simple way to try to make the system Boolean is to just add the constraints to the system. Unfortunately, in that case it is easy to prove that for each and , and of course . It is too easy for SoS to figure out what each should look like. Previously, the variables were unconstrained in any way, and we want to imitate that. We draw inspiration from the Knapsack problem, and we instead replace each instance of the variable with a sum of Boolean variables
and we consider the non-negative polynomial . Clearly there is a degree two proof of non-negativity for this polynomial since we can just replace each instance of with in ( ‣ 5.1).
It remains to show that there are no other proofs that have only small coefficients. Here, we use the fact that the Knapsack problem is hard for SoS: there is no SoS proof of degree less than that is not equal to any number [Gri01]. This allows us to use the Knapsack pseudodistribution to ”pretend” that . Specifically, for each , there is a linear functional defined on polynomials of Boolean variables which satisfies
for any up to degree
for any polynomial up to degree
for any polynomial of degree at most .
Now, take the linear functional defined on each polynomials of variables defined in the following way: Let where is a multiset that contains only the variables corresponding to , and let denote the associated monomial. Then define
Clearly is non-negative on squares and for any up to degree . Because , also satisfies for each and . Finally, because each variable is Boolean, of any monomial is at most one, so for any monomial , . There are at most monomials, so . Just as before, the existence of implies that any degree proof of non-negativity for must contain coefficients of size at least . If we set , then there are variables and no proof of non-negativity with coefficients smaller than doubly-exponential until degree . This proves Theorem LABEL:*thm:counter.
6 Max-Bisection Constraints
In this section, we prove our earlier claim that the Max-Bisection constraints admit rich solutions. Recall the constraints:
Recall that to prove is rich, we have to prove that it is spectrally rich, robust, and complete. Since the solution space lies in the hypercube, it is spectrally rich by Lemma LABEL:*lem:integer-rich, and it is clearly robust since is empty. It remains to prove that it is complete for some . This proof follows a very similar path to [BBCH16], due to the similar symmetry of the constraints.
is -complete for any .
Let denote the solution space of , and let . Any zero eigenvector of can be associated with a polynomial . Since
we must have for each . We argue that any degree polynomial which is identically zero on must have a degree derivation from .
We proceed by induction on . If , the only constant polynomial zero on is the zero polynomial, which has the trivial derivation. Now consider the case of . We proceed in two parts. First, if is fully symmetric, we show that it has a degree derivation. Secondly, for any polynomial which is zero on , we prove that has a degree derivation from , where acts on by permuting the labels of the variables. Taken together, these two facts imply that has a degree derivation from .
To prove the first part, note that a symmetric polynomial is a linear combination of the elementary symmetric polynomials , and it is clear that can be derived by taking the polynomial , reducing it to multilinear using the boolean constraints, and then reducing by for each . This will result in a constant polynomial, which must be the zero polynomial since we are only adding polynomials which are zero on , so the resulting polynomial must be zero on .
To prove the second part, let be the transposition of labels and , and consider the polynomial . Writing , where none of ,,, nor depend on or , we can rewrite
Now because evalutes to zero on any boolean string with exactly ones, if we set and , we know that is a polynomial that must evaluate to zero on any boolean string with exactly ones. Because , by the inductive hypothesis, has a degree proof from (since , clearly ). This implies that has a degree proof from :
where we used the fact that . The degree of this derivation is at most because each has degree at most , and , and similarly for . Thus the inductive hypothesis implies that has a degree derivation, and since transpositions generate the symmetric group, this implies that has a degree proof from . ∎
In this example, is not a Gröbner basis for its ideal . Indeed, the Gröbner basis for this ideal has exponential size. This is an example where our framework is applicable, even though Gröbner bases are intractable to compute.
- [AL94] William Adams and Philippe Loustaunau, An introduction to gröbner bases, American Mathematical Society, 1994.
- [BBCH16] Gábor Braun, Jonah Brown-Cohen, Arefin Huq, Sebastian Pokutta, Prasad Raghavendra, Aurko Roy, Benjamin Weitz, and Daniel Zink, The matching problem has no small symmetric sdp, Proceedings of the Twenty-seventh Annual ACM-SIAM Symposium on Discrete Algorithms (Philadelphia, PA, USA), SODA ’16, Society for Industrial and Applied Mathematics, 2016, pp. 1067–1078.
- [BBH12] Boaz Barak, Fernando G.S.L. Brandao, Aram W. Harrow, Jonathan Kelner, David Steurer, and Yuan Zhou, Hypercontractivity, sum-of-squares proofs, and their applications, Proceedings of the Forty-fourth Annual ACM Symposium on Theory of Computing (New York, NY, USA), STOC ’12, ACM, 2012, pp. 307–326.
- [BRS11] B. Barak, P. Raghavendra, and D. Steurer, Rounding semidefinite programming hierarchies via global correlation, Proc. FOCS, IEEE, 2011, pp. 472–481.
- [BS14] Boaz Barak and David Steurer, Sum-of-squares proofs and the quest toward optimal algorithms, In Proceedings of the 2014 International Congress of Mathematicians. International Mathematical Union (2014).
- [Fol01] Gerald B. Folland, How to integrate a polynomial over a sphere, The American Mathematical Monthly 108 (2001), no. 5, 446–448.
- [Gri01] D. Grigoriev, Complexity of positivstellensatz proofs for the knapsack, computational complexity 10 (2001), no. 2, 139–154.
- [GS11] Venkatesan Guruswami and Ali Kemal Sinop, Lasserre hierarchy, higher eigenvalues, and approximation schemes for graph partitioning and quadratic integer programming with PSD objectives, IEEE 52nd Annual Symposium on Foundations of Computer Science, FOCS 2011, Palm Springs, CA, USA, October 22-25, 2011, 2011, pp. 482–491.
- [GV01] Dima Grigoriev and Nicolai Vorobjov, Complexity of null-and positivstellensatz proofs, Annals of Pure and Applied Logic 113 (2001), no. 1-3, 153–160.
- [Las00] Jean Bernard Lasserre, Optimisation globale et théorie des moments, Comptes Rendus de l’Académie des Sciences-Series I-Mathematics 331 (2000), no. 11, 929–934.
- [Las01] Jean B Lasserre, Global optimization with polynomials and the problem of moments, SIAM Journal on Optimization 11 (2001), no. 3, 796–817.
- [Lau09] Monique Laurent, Sums of squares, moment matrices and optimization over polynomials, Emerging applications of algebraic geometry, Springer, 2009, pp. 157–270.
- [Nes00] Yurii Nesterov, Squared functional systems and optimization problems, High performance optimization, Springer, 2000, pp. 405–440.
- [Oâ17] Ryan OâDonnell, Sos is not obviously automatizable, even approximately, Innovations in Theoretical Computer Science (ITCS) (2017).
- [Par00] Pablo A Parrilo, Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization, Ph.D. thesis, California Institute of Technology, 2000.
- [Sho87] Naum Z Shor, Class of global minimum bounds of polynomial functions, Cybernetics 23 (1987), no. 6, 731–734.