Exploring the Use of Shatter for Through Ramsey-Type Problems

# Exploring the Use of Shatter for Through Ramsey-Type Problems

## Abstract

In the context of SAT solvers, Shatter is a popular tool for symmetry breaking on CNF formulas. Nevertheless, little has been said about its use in the context of AllSAT problems: problems where we are interested in listing all the models of a Boolean formula. AllSAT has gained much popularity in recent years due to its many applications in domains like model checking, data mining, etc. One example of a particularly transparent application of AllSAT to other fields of computer science is computational Ramsey theory. In this paper we study the effect of incorporating Shatter to the workflow of using Boolean formulas to generate all possible edge colorings of a graph avoiding prescribed monochromatic subgraphs. Generating complete sets of colorings is an important building block in computational Ramsey theory. We identify two drawbacks in the naïve use of Shatter to break the symmetries of Boolean formulas encoding Ramsey-type problems for graphs: a “blow-up” in the number of models and the generation of incomplete sets of colorings. The issues presented in this work are not intended to discourage the use of Shatter as a preprocessing tool for AllSAT problems in combinatorial computing but to help researchers properly use this tool by avoiding these potential pitfalls. To this end, we provide strategies and additional tools to cope with the negative effects of using Shatter for AllSAT. While the specific application addressed in this paper is that of Ramsey-type problems, the analysis we carry out applies to many other areas in which highly-symmetrical Boolean formulas arise and we wish to find all of their models.

## 1Introduction

The AllSAT problem is a variant of the SAT problem where we are interested in finding all the models (satisfying assignments) of the input formula. Applications of AllSAT to fields like model checking [28] and data mining [14] have drawn attention to AllSAT solvers. The survey by Toda and Soh [25] summarizes the state-of-the-art in techniques used for solving AllSAT problems. Many ideas from SAT solvers are applicable to AllSAT solvers with few adaptations. For example, the basic idea behind AllSAT solvers based on blocking-clauses [21] is to iteratively find a model using a conventional SAT solver and then add a clause that blocks that model, until no models are found. This intuitive idea serves as a base for other more advanced approaches to AllSAT solvers [29]. A technique that has proved effective in SAT solvers and is of particular importance for AllSAT problems is symmetry breaking in Boolean formulas [24]. Much research has been devoted to develop methods that prevent SAT solvers from exploring redundant search spaces in search for a satisfying assignment. The survey by Walsh [26] provides a good overview of the current approaches and techniques used to deal with this problem. Extensions of symmetry breaking for SAT have been studied for different related problems like MaxSAT [15] and #SAT [16], and in the context of Answer Set Programming [8]. Shatter [3] is a tool that generates symmetry-breaking clauses as a preprocessing step to solve Boolean formulas in order to simplify the search space for conventional SAT solvers. Shatter has become a popular preprocessing tool since it can be used on any CNF formula encoded in the popular DIMACS format and it has the desirable property that the size of the symmetry-breaking clauses it adds to the formula is linear in the number of variables of the original formula.

To study the effect of Shatter’s symmetry breaking approach in a clean application where symmetries can be formally defined and studied, we look at symmetry breaking for Ramsey-type problems in combinatorial computing. Ramsey-type problems are those related to conditions by which it is impossible to avoid prescribed substructures when partitioning the set of elements of a discrete object. Computational approaches to solve these kinds of problems have recently leveraged advances on SAT solvers [12]. In this paper we focus on graph 2-colorings, which are partitions of the set of edges of a graph into two sets. We say that a graph arrows the pair of graphs , written , when any 2-coloring of the edges of a graph yields a monochromatic in the first color or a monochromatic in the second color.

It is straightforward to see how 2-color arrowing problems can be encoded into Boolean satisfiability problems. In the case of finite Ramsey numbers, a description of this encoding appears in Zhang’s chapter in the Handbook of Satisfiability [30]. Boolean formulas arising from this encoding have a prominent place in complexity theory because they provide hard examples for resolution. The complexity of resolution of such formulas was studied by Krishnamurthy [18], Krajíček [17], and Pudlák [22], among others.

On the one hand, determining the satisfiability of a Boolean formula encoding (i.e., solving the SAT problem for this formula) can answer the question of whether arrows the pair . On the other hand, finding all the assignments that satisfy the formula (i.e., solving the AllSAT problem for this formula) can be used to generate the complete set of colorings of the edges of witnessing . Generating these sets of colorings for one parameter is often used as a building block towards finding exact values for Ramsey numbers under different parameters. The “gluing method” used to establish [19], which is still found embedded in more recent ideas [5], is an example of such an application. Using AllSAT solvers for this purpose offloads the combinatorial search to standard tools that are being actively developed [25], eliminating the need to craft specialized code. A shortcoming of generating these families through the method of encoding the negation of the arrowing property into a Boolean formula is that an AllSAT solver may generate many colorings that are essentially equivalent among them as it lists all possible models of the formula. The need to generate models that are distinct under some notion of equivalence is not unique to formulations of the arrowing predicate as a Boolean formula. Many symmetry breaking techniques have been developed for specific applications of SAT. A great example of a symmetry breaking technique specifically tailored for graph search is the use of canonizing sets for small graph searches [13]. Unfortunately, the need to embed these techniques into the Boolean encoding of the non-arrowing property may neglect the advantage of using general-purpose AllSAT solvers. To avoid this issue, one may be interested in using off-the-shelf symmetry breaking software that is domain independent.

Given Shatter’s popularity, it may seem as a good tool to tackle the generation of irredundant sets of colorings for Ramsey-type problems. Nevertheless, in this paper we identify and discuss some drawbacks of using this approach without caution. One of these drawbacks stems from the fact that Shatter’s main goal is to avoid adding symmetry-breaking clauses whose number of literals is quadratic in the number of variables while making it easier to find a (not every) satisfying assignment of the original formula should one exist. This is done at the expense of preserving the number of “unique” satisfying assignments of the original formula. Another drawback stems from the fact that Shatter works directly at the Boolean formula level to detect symmetries, so it has no access to domain-specific symmetries that are not carried over by the encoding. The issues presented in this work are not intended to discourage the use of Shatter as a preprocessing tool for AllSAT problems in combinatorial computing but to help researchers properly use this tool by avoiding these potential pitfalls. To this end, we provide strategies and additional tools to cope with the negative effects of using Shatter for AllSAT.

The rest of this paper is organized as follows. In Section 2 we review Shatter’s approach to symmetry breaking and give a brief background on Ramsey-type problems. In Sections Section 3 and Section 4 we address the problems of the number of satisfying assignments of the resulting formulas after preprocessing with Shatter and the generation of incomplete sets of colorings, respectively. Finally, Section 5 adds some concluding remarks and directions for future work.

## 2Background and Definitions

Shatter is a domain-independent preprocessing tool [3] which is available online1 and works on Boolean formulas expressed in the standard DIMACS format, making it very convenient to integrate in any existing workflow. A fundamental construction in Shatter’s formulation of symmetry breaking predicates is a graph that encodes the relationship between clauses and literals of the input formula [2]. This graph is defined as follows:

The symmetry breaking predicates added by Shatter come from the automorphisms of the graph . Recall the automorphisms of a graph are the permutations of its vertices that leave its edge set unchanged, and that these form a group. In the case of vertex-colored graphs, these permutations respect the coloring in the sense that no vertex is mapped to a vertex of a different color. A bird’s eye view of the process Shatter uses to break the symmetries of a Boolean formula is as follows:

.

Let be a permutation in the automorphism group of , and consider the restriction of to the set of literal vertices of . is a permutation of the literals of because the color of the literal vertices is different from that of the clause vertices. Let be an ordering of the variables of and let with . Then the key observation behind lexicographical symmetry breaking is that

for if is a satisfying assignment of , is a satisfying assignment of as well, and one of and will precede the other in lexicographical order (considering the strings for ) thus satisfying the additional constraints. Following the same principle, one can prove that adding lexicographical constraints for other permutations in the automorphism group of will also preserve satisfiability while reducing the number of models of the resulting formula. Notice that this formulation of symmetry breaking adds expressions involving subexpressions, but itself has a literal count that is linear in , so the number of literals added per automorphism is quadratic in the number of variables of .

This set of rules corresponds to the most basic version of lexicographical symmetry breaking. Shatter implements several optimizations over this basic workflow which have desirable effects on determining the satisfiability of the input formula. In Sections Section 3 and Section 4, on the other hand, we study the effect of using this tool for an AllSAT application: reducing the number of models found through encodings of the non-arrowing property as a Boolean formula. We define the concepts from Ramsey theory that are relevant to our analysis in the following section.

### 2.1Ramsey Colorings

For a graph , let and be the set of vertices and edges of , respectively. The order of a graph is the number of vertices of the graph. For two graphs and , we denote by the set of (not necessarily induced) subgraph isomorphisms from to . More formally, the set is the set of injective functions from to such that if then .

An edge coloring of is a function from the set to a finite set of colors. If the cardinality of the set of colors is , the coloring is said to be a -coloring. Two edge colorings of a graph are isomorphic if one can be turned into the other by permuting the vertices of , i.e., two colorings and are isomorphic if there exists a permutation of such that for every edge , . We define a “deduplication” operation on sets of colorings that preserves only one representative of each of the equivalence classes in the input set of colorings. Formally, for a set of colorings of a graph , is defined as a set of colorings such that implies is not isomorphic to , and for every there exists a such that is isomorphic to . The choice of which representative is kept by this operation is not crucial to our application, so we can assume it is the minimum over some ordering of the colorings in . A desirable property of is that it can be used to recover the full set of colorings so it is an efficient way to store the entire set .

For graphs , and , we say the graph arrows the pair if every 2-coloring of the edges of either contains a monochromatic in the first color or a monochromatic in the second color. We denote this property by . A 2-coloring of the edges of that contains no monochromatic in the first color and no monochromatic in the second color is said to witness the fact that . The most classical example of arrowing is the fact that the edges of the complete graph on 6 vertices cannot be 2-colored in a way that avoids monochromatic triangles, so . The edges of , on the other hand, can be colored as shown in Figure 1 avoiding monochromatic triangles, so . Several Ramsey-type problems can be expressed in terms of the arrowing property: the generalized Ramsey number can be defined as the smallest natural number such that arrows the pair 2, while the generalized (edge) Folkman number , which is another concept commonly studied in Ramsey theory, can be defined as the smallest order of a -free graph that arrows the pair . We define as the set of witness colorings of . Clearly, . We will call a set of colorings for parameters , and complete3 if every isomorphism class in is represented in the set. For instance, itself is trivially complete, and is complete by definition.

To use AllSAT solvers to generate graph 2-colorings, we exploit the fact that the Boolean domain contains two values (true) and (false) and express the negation of the arrowing property in terms of Boolean formulas. Consider the Boolean formula on variables for defined as per equation ( ?) in Figure ?. This formula essentially states that in every subgraph isomorphism from to at least one of the edges involved is “colored” true, and in every subgraph isomorphism from to at least one of the edges involved is “colored” false. If we define the set of colors as , then a model of can easily be mapped to an edge coloring of that avoids monochromatic copies of and in the first and second color, respectively. It is also easy to see this mapping is one-to-one, i.e., to each model of corresponds a coloring witnessing and vice versa. Thus, . Furthermore, one can generate by using an AllSAT solver to list every model of . As mentioned before, an undesirable characteristic of this approach is that can be large and one is in general more interested in generating the smaller set directly from the SAT formulation of .

## 3Number of Satisfying Assignments

One of the main improvements of Shatter [3] over the original formulation of the symmetry breaking clauses [6] explained in Section 2 is that Shatter adds symmetry-breaking clauses whose number of literals is linear in the number of variables of the input formula. This is done through a relaxation on the symmetry breaking constraints. This relaxation has an undesirable effect in the number of satisfying assignments of the resulting formula. To study this effect, we summarize some of the details behind Shatter’s relaxation of the symmetry breaking constraints.

Using additional equality variables , one incurs in a quadratic increase on the length of the formula when adding lexicographical symmetry breaking clauses, as mentioned in Section 2. Shatter avoids this by using “chaining predicates”. For this, new variables (that is, is true if and only if is “less than or equal to” ) and are introduced, together with clauses , with and . Shatter also replaces equality variables with “greater than or equal to” variables , and relaxes the if and only if clauses for the variables to one way implications. The clauses added by Shatter are then the CNF equivalents of formulas of the form , with . It is easy to see these relaxations introduce satisfying assignments that do not satisfy the original symmetry breaking formulation. To get a better feel of how much of a “blow-up” in the number of satisfying assignments does this relaxation cause, consider the following lemmas about extensions of partial assignments of the Boolean expressions involved in these two formulations of symmetry breaking clauses. in Lemma ? corresponds to the original lexicographical symmetry breaking clauses with chaining, while in Lemma ? corresponds to the clauses added by Shatter.

Let , that is, is without fixing . Then . We first prove by induction in that a partial assignment of the variables and has at most one extension that satisfies once is assigned to a value. For , this is trivially true since the value of does not depend on the value of . We now assume that this property holds for and prove it for . Our induction hypothesis implies that for any assignment of the variables and for , there is at most one assignment of the variables , that satisfies once is assigned to a value. Suppose we assign to a value, then since is a subformula of so is uniquely determined by the assignments of , and , which proves our claim. Since , any satisfying assignment of must fix to and the lemma follows.

Since assigns to , the subformula simplifies to , regardless of what and are assigned to. Then can be assigned to or without falsifying . Furthermore, does not constrain if is assigned to , and simplifies to when is assigned to , so can be assigned to or as well. Following the same reasoning, all the variables can be assigned to or independently without falsifying except possibly for if , yielding the desired result.

The combination of these lemmas means that, in the worst case, the increase in the number of satisfying assignments due to Shatter’s relaxations may be exponential in the number of or variables. While in the original lexicographical symmetry breaking formulation the number of variables is the same as the number of variables in the original formula, Shatter reduces the number of variables to consider by eliminating certain tautological subformulas from , thus mitigating the effect of the increase in the number of models.

Another important detail in the analysis above is that the “blow-up” in the number of models of the formula enhanced with symmetry breaking clauses happens only at the variables: for a fixed assignment of the original variables of there are many possible assignments of the variables that would satisfy the formula output by Shatter. The projective model enumeration technique [9] in the clasp solver [10] is particularly useful to nullify this increase in the number of models. This technique allows for outputting models that are different modulo a subset of the variables. Thus combining projective model enumeration with the output of the formula preprocessed by Shatter restores the original goal of symmetry breaking which is reducing the number of satisfying assignments. The subset of variables that one would project to would of course be the original set of variables of the input formula.

In Section 5 we point at BreakID [7], another symmetry breaking tool, which avoids this issue altogether by using the original symmetry breaking predicates without relaxations. This tool is thus more appropriate for AllSAT applications where symmetry breaking is needed.

### 3.1An Example: Extra close brace or missing open brace

To illustrate this issue we provide a concrete example. From finite Ramsey theory, we know that [4], where is the cycle of length 5 (see also [23] for a comprehensive survey of what is known in finite Ramsey theory). This means that , but , so we are interested in finding all edge colorings of the complete graph witnessing . contains 28 variables (corresponding to edges in ) and 1344 clauses, and there are 1190 models for that formula. From this information, we know that . After processing this formula with Shatter, the resulting formula with symmetry breaking clauses has 70 variables, 1499 clauses, and 824 models. On the other hand, using our own implementation of the chaining method without the relaxation outputs a formula with 165 variables, 1809 clauses, and 5 models. Using nauty [20] to reduce any of these sets of colorings to pick just one representative from each equivalence class of colorings under isomorphism, we find that , so the chaining method without the relaxation outputs only one redundant coloring.

## 4Incomplete Sets of Colorings

Perhaps more concerning than the increase in the number of colorings found by preprocessing the formula is the fact that enumerating all models of the result of preprocessing with Shatter may not yield a complete (according to our definition in Section 2) set of colorings. On the other hand, there are parameters , , and for which preprocessing a formula with Shatter does output a Boolean formula whose models can be used to build a complete set of colorings for the given parameters. One example was presented in Section 3.1, where we were able to generate from the models of after preprocessing it with Shatter.

The fact that for certain parameters this method will work and for other parameters it will not warrants an investigation. In this section, we take a closer look at this phenomenon and present a sufficient condition under which a workflow that uses Shatter for symmetry breaking produces incomplete sets of colorings. This condition is the presence of free variables, which are variables that do not appear in the CNF formula. This is possible in the context of CNF formulas in the DIMACS format because the specification4 states that “it is not necessary that every variable appear in the instance.” We study the effect of free variables in Lemma ? and as an immediate consequence, we state a sufficient condition for to produce incomplete sets of colorings in Theorem ?.

Let be the set of free variables in . The graph generated according to the rules summarized in Section 2 has 2 vertices for each free variable (one for the positive literal and one for the negative literal) and an edge between these two vertices. Then the matching on vertices (i.e., the disjoint union of edges) is an induced subgraph of . From graph theory, we know that the permutations for are generators of the automorphism group of (see, for instance, [27], Theorem 3-11). The rules applied by Shatter will turn these permutations into clauses of the form for , so each variable in will be assigned to .

The above proof is based on the rules presented in [3] to generate the symmetry breaking clauses. In practice, version 0.3 of Shatter5 seems to have an additional rule that assigns each free variable to its own color. This change is effective in eliminating permutations of the type for from the set of generators of the automorphism group of (because and will now have different colors), yet it does not eliminate permutations of the type which are the culprit of Lemma ?. It is important to highlight the fact that the addition of single-clause variables (a clause is equivalent to ) is not a flaw in the design of the symmetry breaking clauses in Shatter but a feature, as noted in Section 3 of [1]. This is because processing permutations this way reflects the fact that free variables in the input formula can be fixed to any Boolean value, in this case , which is advantageous when trying to determine the satisfiability of the input formula.

As an immediate consequence of Lemma ?, we have the following theorem.

Let be the formula output by preprocessing with Shatter. Because does not participate in any subgraph isomorphism from to , or from to , the variable is free in . By Lemma ? it will be assigned to in any model of . Let be a model of the formula and let be the assignment obtained from by assigning to . Because is free in , the restriction of to the variables of (i.e., the restriction of to the original variables) is a model of , but itself is not a model thus the coloring corresponding to will not be represented in the colorings obtained from the models of .

An obvious modification one can do to avoid this issue is to restrict the set of variables in equation ? in Figure ? to for edges that are actually involved in some subgraph isomorphism from to or from to . Unfortunately, this does not guarantee the resulting formula will output a complete set of colorings. There are cases where the models of formulas encoding the non-arrowing property do not generate complete sets of colorings even when the formula does not have free variables. This indicates Theorem ? is not a necessary condition for this phenomenon to occur. Here we provide an example of such a formula, which is related to the example presented in Section 3.1: Let be the graph obtained from by selecting one of its vertices and removing all but 2 edges incident to it (alternatively, this graph can be obtained by adding a vertex to and connecting it to two of the original vertices). The resulting graph is illustrated in Figure 2. We are interested in obtaining all 2-colorings avoiding monochromatic in any of the colors. It is easy to see that has no free variables: because the only vertex of degree less than 6 has degree 2, it participates in at least one cycle of length 5. Nevertheless, the models of the formula obtained from preprocessing with Shatter represent 64 of the 90 possible isomorphism classes in .

## 5Conclusion and Related Work

We looked at the use of Shatter, a popular tool for symmetry breaking of CNF formulas, in the context of AllSAT problems. Specifically, we presented and discussed two issues related to applying Shatter to formulas encoding the non-arrowing property. The first issue was related to the number of satisfying assignments, showing that Shatter may incur in an increase in the number of models found. We also mentioned the projective enumeration feature in clasp as a tool to deal with this issue. The second issue we discussed was related to the completeness of the set of colorings generated from the Boolean formula after preprocessing with Shatter. We presented a sufficient condition for the set of colorings obtained this way to be incomplete. A direction for future work in this area is completing this analysis by providing necessary conditions.

While Shatter has been an influential tool in the field of symmetry breaking in Boolean formulas for over a decade, it is not the only tool available as a drop-in addition to existing workflows. Recently, BreakID [7] has built upon the symmetry breaking techniques introduced by Shatter and has added some novel ideas like row interchangeability. Even though BreakID implements the same relaxations as Shatter, it does includes an option to no use these relaxations and is thus better suited for AllSAT applications since it will not introduce additional models.

## Acknowledgments

The author would like to thank Dr. Edith Hemaspaandra and Dr. Stanisław P. Radziszowski at the Rochester Institute of Technology, as well as the anonymous reviewers of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) for their valuable comments and suggestions for improvements.

### Footnotes

1. http://www.aloul.net/Tools/shatter/
2. The “classical” Ramsey numbers are defined in terms of generalized Ramsey numbers as for positive integers and .
3. Not to be confused with other uses of completeness in the context of symmetry breaking for Boolean formulas. For instance, [13] defines the concepts of sound and complete symmetry breaking predicates to compare the characteristics of different types symmetry breaking predicates, but in our context we have fixed the type of symmetry breaking predicate to lexicographical symmetry breaking.
4. ftp://dimacs.rutgers.edu/pub/challenge/satisfiability/doc/satformat.dvi
5. http://www.aloul.net/Tools/shatter/Shatter_Linux_v03.tar.gz

### References

1. Solving difficult SAT instances in the presence of symmetry.
F. A. Aloul, A. Ramani, I. L. Markov, and K. A. Sakallah. In Proceedings of the 39th annual Design Automation Conference, pages 731–736. ACM, 2002.
2. Solving difficult instances of boolean satisfiability in the presence of symmetry.
F. A. Aloul, A. Ramani, I. L. Markov, and K. A. Sakallah. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 22(9):1117–1137, 2003.
3. Efficient symmetry breaking for boolean satisfiability.
F. A. Aloul, K. A. Sakallah, and I. L. Markov. IEEE Transactions on Computers, 55(5):549–558, 2006.
4. On the existence of specified cycles in complementary graphs.
G. Chartrand and S. Schuster. Bulletin of the American Mathematical Society, 77:995–998, 1971.
5. Computing the Ramsey number R(4, 3, 3) using abstraction and symmetry breaking.
M. Codish, M. Frank, A. Itzhakov, and A. Miller. Constraints, 21(3):375–393, 2016.
6. Symmetry-breaking predicates for search problems.
J. M. Crawford, M. L. Ginsberg, E. M. Luks, and A. Roy. In L. C. Aiello, J. Doyle, and S. C. Shapiro, editors, International Conference on the Principles of Knowledge Representation and Reasoning, pages 148–159. Morgan Kaufmann, 1996.
7. Improved static symmetry breaking for SAT.
J. Devriendt, B. Bogaerts, M. Bruynooghe, and M. Denecker. In N. Creignou and D. L. Berre, editors, International Conference on Theory and Applications of Satisfiability Testing, volume 9710 of Lecture Notes in Computer Science, pages 104–122. Springer, 2016.
8. On local domain symmetry for model expansion.
J. Devriendt, B. Bogaerts, M. Bruynooghe, and M. Denecker. Theory and Practice of Logic Programming, 16(5-6):636–652, 2016.
9. Solution enumeration for projected boolean search problems.
M. Gebser, B. Kaufmann, and T. Schaub. In Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, pages 71–86. Springer, 2009.
10. Conflict-driven answer set solving: From theory to practice.
M. Gebser, B. Kaufmann, and T. Schaub. Artificial Intelligence, 187-188:52–89, 2012.
11. Memory efficient all-solutions SAT solver and its application for reachability analysis.
O. Grumberg, A. Schuster, and A. Yadgar. In A. J. Hu and A. K. Martin, editors, International Conference on Formal Methods in Computer-Aided Design, volume 3312 of Lecture Notes in Computer Science, pages 275–289. Springer, 2004.
12. The science of brute force.
M. J. H. Heule and O. Kullmann. Communications of the ACM, 60(8):70–79, 2017.
13. Breaking symmetries in graph search with canonizing sets.
A. Itzhakov and M. Codish. Constraints, 21(3):357–374, 2016.
14. Boolean satisfiability for sequence mining.
S. Jabbour, L. Sais, and Y. Salhi. In Proceedings of the 22Nd ACM International Conference on Information & Knowledge Management, CIKM ’13, pages 649–658, New York, NY, USA, 2013. ACM.
15. Lifted symmetry detection and breaking for map inference.
T. Kopp, P. Singla, and H. Kautz. In Advances in Neural Information Processing Systems, pages 1315–1323, 2015.
16. Toward caching symmetrical subtheories for weighted model counting.
T. Kopp, P. Singla, and H. A. Kautz. In A. Darwiche, editor, AAAI Workshop: Beyond NP, volume WS-16-05 of AAAI Workshops. AAAI Press, 2016.
17. On the weak pigeonhole principle.
J. Krajíček. Fundamenta Mathematicae, 170:123–140, 2001.
18. Examples of Hard Tautologies and Worst-case Complexity Results for Propositional Proof Systems.
B. Krishnamurthy. PhD thesis, 1981.
19. .
B. McKay and S. Radziszowski. Journal of Graph Theory, 19(3):309–322, 1995.
20. Practical graph isomorphism, II.
B. D. McKay and A. Piperno. Journal of Symbolic Computation, 60:94–112, 2014.
21. Applying SAT methods in unbounded symbolic model checking.
K. L. McMillan. In E. Brinksma and K. G. Larsen, editors, International Conference on Computer Aided Verification, volume 2404 of Lecture Notes in Computer Science, pages 250–264. Springer, 2002.
22. A lower bound on the size of resolution proofs of the Ramsey theorem.
P. Pudlák. Information Processing Letters, 112(14-15):610–611, 2012.
23. Small Ramsey numbers.
S. Radziszowski. Electronic Journal of Combinatorics, Dynamic Surveys, DS1, 1994.
24. Symmetry and satisfiability.
K. A. Sakallah. In A. Biere, M. Heule, H. van Maaren, and T. Walsh, editors, Handbook of Satisfiability, volume 185 of Frontiers in Artificial Intelligence and Applications, pages 289–338. IOS Press, 2009.
25. Implementing efficient all solutions SAT solvers.
T. Toda and T. Soh. Journal of Experimental Algorithmics, 21:1.12:1–1.12:44, Nov. 2016.
26. Symmetry breaking constraints: Recent results.
T. Walsh. In J. Hoffmann and B. Selman, editors, Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, pages 2192–2198. AAAI Press, 2012.
27. Graphs, Groups and Surfaces.
A. T. White. North-Holland Pub, 1973.
28. Hybrid BDD and All-SAT method for model checking.
A. Yadgar, O. Grumberg, and A. Schuster. In Languages: From Formal to Natural, pages 228–244. Springer, 2009.
29. All-SAT using minimal blocking clauses.
Y. Yu, P. Subramanyan, N. Tsiskaridze, and S. Malik. In Proccedings of the 27th International Conference on VLSI Design and 13th International Conference on Embedded Systems, pages 86–91, Jan 2014.
30. Combinatorial designs by SAT solvers.
H. Zhang. In Handbook of Satisfiability, volume 85 of Frontiers in Artificial Intelligence and Applications, pages 533–568. IOS Press, 2009.
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters