Abstract
Optimization of MixedInteger NonLinear Programming (MINLP) supports important decisions in applications such as Chemical Process Engineering. But current solvers have limited ability for deductive reasoning or the use of domainspecific theories, and the management of integrality constraints does not yet exploit automated reasoning tools such as SMT solvers. This seems to limit both scalability and reach of such tools in practice. We therefore present a tool, ManyOpt, for MINLP optimization that enables experimentation with reduction techniques which transform a MINLP problem to feasibility checking realized by an SMT solver. ManyOpt is similar to the SAT solver ManySAT in that it runs a specified number of such reduction techniques in parallel to get the strongest result on a given MINLP problem. The tool is implemented in layers, which we may see as features and where reduction techniques are feature vectors. Some of these features are inspired by known MINLP techniques whereas others are novel and specific to SMT. Our experimental results on standard benchmarks demonstrate the benefits of this approach. The tool supports a variety of SMT solvers and is easily extensible with new features, courtesy of its layered structure. For example, logical formulas for deductive reasoning are easily added to constrain further the optimization of a MINLP problem of interest.
ManyOpt: An Extensible Tool for Mixed, NonLinear Optimization Through SMT Solving
Andrea Callia D’Iddio and Michael Huth
Department of Computing, Imperial College London
London, SW7 2AZ, United Kingdom
a.calliadiddio14, m.huth@imperial.ac.uk
1 Introduction
MINLP problems are like the familiar linear programming models, except that constraints or the objective function may contain nonlinear terms, and that some variables may not have realvalued type but an integral or even binary one. The underlying decision problem is undecidable in general. But solving MINLP problems is crucial in many areas of application, such as Chemical Process Engineering. Current mathematical tools for solving MINLP problems, for example ANTIGONE [34], made important progress in solving problems that occur in practice. But they presently do not support deductive reasoning and incremental solving – which could generate explanatory scenarios and support “whatif” questions. They also don’t yet exploit the powerful reasoning techniques of SMT solvers in the transformation of MINLP problems into problem types that offer computational advantages – such as NLP whose problems have no integrality constraints.
Stateoftheart SMT solvers do support deductive reasoning and incremental solving, but they usually have little support for nonlinear arithmetics and arithmetics that combine real and integer types, as is the case in MINLP problems. This motivated us to write a tool, called ManyOpt, whose purpose is to be an extensible workbench for exploring and evaluating how MINLP problems can be reduced to instances of SMT solving. The tool therefore inherits the ability of incremental solving and deductive reasoning from SMT solvers.
The default mode of our tool executes in parallel different reduction techniques from MINLP to instances of SMT solving. This parallelization means that these techniques compete on a given input and so may speed up solving since the “winner” will terminate optimization. Reduction techniques themselves are conceptualized as feature vectors, and so are a composition of features that may coordinate or complement different reduction activities. Our tool can therefore be extended with new features and new feature vectors to get richer parallelized solvers for MINLP problems. The tool makes the assumption that optimal values are only computed up to some specified accuracy, a further input parameter to the tool.
Reduction techniques need to deal with computational problems related to the management of integrality constraints and to the nonlinearity of terms. The tool ManyOpt deals with such integrality management by using the parallel combination of dynamic branch and bound techniques and a novel static binarized flattening technique. The experiments reported in this paper demonstrate that the combination of such techniques is instrumental in solving many MINLP benchmarks that would otherwise not be solvable within reasonable running times by branch and bound techniques that reduce to SMT problem instances.
The nonlinearity of terms is dealt with by the parallel combination of a naive search technique, which has shown to often work well with satisfiability solving (see MiniSAT+ [18]), with an unbounded binary search technique, which is faster in scanning intervals containing many feasible objective values. Our tool also supports the combination of these two techniques into a hybrid approach whose cooperative strategy conjoins the invariants of both techniques. Our experiments provide evidence that the approach we advocate and develop here has definite potential in crafting MINLP solvers based on SMT.
Contributions of our paper:
We offer a tool with which researchers can explore techniques for reducing MINLP optimization problems to instances of SMT, where optimization is computed up to some specifiable accuracy. We organize the design space for such reduction techniques such that points in that space are feature vectors, where features are choices within a particular reduction layer of that tool. These layers are easily extensible with new features and the tool supports, in principal, the use of any SMTLIB 2.0 compliant SMT solver. The presently implemented features of ManyOpt represent some techniques already familiar in the optimization community but also novel techniques that are made possible by using an SMT solver as a feasibility checker. We evaluate this tool through standard MINLP benchmarks. In particular, we show that our tool can solve some benchmarks from Chemical Process Engineering that are widely held to be challenging for existing MINLP solvers: our tool solves all these problems and with higher accuracy than that reported in some existing MINLP tools. We also evaluate ManyOpt on 193 benchmarks from the MINLPLIB library, 138 NLP problems and 56 MINLP problems. Our tool solved 68% of the MINLP problems and 63% of the NLP problems within 30 minutes. This appears to be the strongest evidence yet of the potential of SMT solving for MINLP problems. The analysis of these experiments also clearly reveals the benefits of running different reduction techniques in parallel. For example, using just one reduction technique for dealing with nonlinearity would dramatically reduce the percentage of solved benchmarks and increase the computing time for solving.
Outline of paper
In Section 2 we recall some background on MINLP problem types and SMT. The tool architecture and functionality of ManyOpt is the topic of Section 3. A description of the algorithms and supported features is given in Section 4. In Section 5 we evaluate our approach and tool experimentally. Related work is discussed in Section 6 and the paper concludes in Section 7.
2 Background
Mixed Integer Nonlinear optimization. A Mixed Integer NonLinear Programming (MINLP) problem is a mixed integer optimization problem where the objective function, some constraints or both may be nonlinear [19]. This leads to established subtypes of MINLP: In LP and NLP problems, there are no integrality constraints. In LP, the objective function and the constraints are linear, in NLP problems they can also be nonlinear. ILP and INLP problems are variants of LP and NLP, respectively, in which all variables must have integer values. MILP is a subtype of MINLP in which constraints and objective function are linear. BLP, BNLP, MBLP and MBNLP correspond to ILP, INLP, MILP, and MINLP, respectively, but integrality constraints are replaced by “binarity” constraints: all variables from the designated set must have binary instead of integral values.
Complexity analysis. Unfortunately, the problem of solving a generic MINLP problem to deterministic global optimality is undecidable [25], due to a combination of nonlinearity and integrality. Notwithstanding, several efforts have been made to solve MINLP problems, supported by decomposition or transformation techniques specific to subclasses of MINLP: Solving a generic NLP problem has been proven to be NPhard [36], but many effective methods and algorithms have been developed in the last four decades (see [3] for a survey). Solving a generic ILP or MILP problem is also NPhard [13, 26], but again much progress has been done to develop effective techniques (see [13] for example).
Optimization based on SMT. Satisfiability Modulo Theories (SMT) solvers [16] are an extension of SAT solvers, where additional theories can be used to express logical formulae and to find a satisfying assignment. The SMT solvers contain specific algorithms based on many different theories (see e.g. [29]), including: theory of equality, bitvectors, reals, integers, and arrays. An important and popular SMT solver among the ones supported directly in our tool is Z3 [15]. Z3 is an efficient incremental SMT solver that supports exact arithmetic [8], which is fundamental whenever exact solutions or high confidence in approximate solutions is required.
The theories modeled in SMT and the related languages provide support to naturally encode the variables and the constraints of an optimization problem, thus SMT solvers can be used as a black box in order to act as a feasibility solver for optimization algorithms.
In [7] the Z3 solver has been extended in order to also perform optimization. The resulting solver, called Z, can be used to combine optimization with the powerful capabilities of Z3. The Z solver has shown the ability to efficiently solve many MILP optimization problems and it competes well with other SMTbased optimizers presented in [40] and [31] in this area. But Z supports only linear mixed arithmetic, thus it has very little support for MINLP problems as such. The main purpose of our research is to add such capability in the context of SMTbased optimization.
3 Tool Architecture and Functionality
Figure 1 shows the main structure of our tool ManyOpt, represented as a data flow diagram. As input it takes a desired accuracy for the computed global optimum and, either an OSiL model for a MINLP problem or an MPS model for a MILP problem which is translated into OSiL. Then ManyOpt executes in parallel a set of dataflows on that OSiL input, where each dataflow has its own feature vector of how to realize configurable tool layers. The tool layers are Preprocessing, Integrality Management, Continuous Relaxation Optimization, and Feasibility Checking. Each of these layers contains within it an (extensible) list of features, as seen in Fig. 1. The default setting uses 18 feature vectors run as parallel processes and keeps track of dependencies within each such vector. For example, if feature BinarizedFlattening is enabled in Preprocessing, then feature Disabled has to be chosen for Integrality Management.
For each feature vector, the data flow of its process is as follows. First, an SMTLIB 2.0 representation of the model is constructed, based on the input and selected features. After that, a Preprocessing phase may transform the SMTLIB 2.0 representation using either Binarization or our novel Binarized Flattening techniques to convert the MINLP problem into a MBNLP problem, where all integer variables can only have values 0 or 1.
Second, the Main Optimization Process uses the lower three layers, organized as a stack, to find an optimal solution for the given model within the specified accuracy. The Integrality Management layer may apply branchandbound techniques such as OneByOne or our novel AllInOne to handle integrality constraints. This layer produces NLP problems as continuous relaxations of the original MINLP problem, and then relies on the Continuous Relaxation Optimization layer to solve them. Alternatively, Integrality Management may be disabled, for example if the original problem is already an NLP problem, or if we wish to move the complexity of integrality management to lower layers.
The Continuous Relaxation Optimization layer takes an NLP problem as input and reduces optimization to the repeated use of the Feasibility Checking layer. A number of such reduction algorithms are supported: the naive method (the term “naive” is used without prejudice and inspired by work on pseudoboolean optimization [18]), the unbounded binary search method, and a novel method of us based on a hybrid combination of these two methods.
The feasibility of an NLP problem, represented in SMTLIB 2.0 in our tool, is verified using an SMT solver. Currently, ManyOpt supports in the feasibility layer in Figure 1, the solver Z3 [15] directly, the solvers MathSAT [10], CVC4 [2] and YICES [17] through the PySMT library [23]; other SMT solvers compatible with SMTLIB 2.0 may be used through a POSIX piping.
4 Tool description: algorithms and software design
4.1 Input parsing: OSiL and MPS
Our ManyOpt tool supports two input formats for optimization models. The OSiL format [20] has been defined and published by COINOR within their Optimization Services project [37]. It is a widely adopted XML representation (e.g. in [11]) of an optimization problem, and its schema is available in [38].
The OSiL language distinguishes among linear constraints, quadratic constraints and generic nonlinear constraints. Linear constraints are represented using a compact coefficient matrix, and quadratic constraints are represented by a tuple list. Nonlinear constraints are represented by using trees, where each node represents an operator and its children represent the operands.
Using an XML representation for MINLP problems has several benefits in terms of robustness, stability and standardizability. In particular, using an XML schema facilitates checking for errors in an OSiL model without the need to write code for this purpose. Given an OSiL model as input, ManyOpt constructs several alternative SMTLIB 2.0 representations of it, whose construction is based on the configuration values in each feature vector corresponding to a specific parallel execution.
Our tool also supports Mixed Integer Linear Programming models (MILP) represented in MPS format, which is another wellknown format for optimization (see e.g. [28]), still consisting of a compact coefficient matrix representing linear constraints. We added this support by writing a simple C++ code entirely based on the conversion libraries provided by COINOR, in order to convert the MPS input into OSiL format.
Our tool OSiL parser can detect whether the problem is MINLP or just NLP, and in the second case it can enable in ManyOpt only those feature vectors that are useful for NLP, thus saving system resources and improving performance.
4.2 Preprocessing: binarization and flattening
Binarization Binarization mechanisms are widely used for mixed optimization, in order to handle integrality constraints where variables can only have values 0 or 1 (see [19] Subsection 6.2.1), thus transforming the MINLP problem into an MBNLP problem. In order to achieve this, each variable involved in an integrality constraint is represented by creating a set of variables representing the “bits” of the original variable, which is assumed to be bounded in order to determine the amount of “bits” needed. Then and are set as lower and upper bounds for these new variables, and they are constrained to be integer, i.e. either or .
Formally, an integrality constraint for variable bounded by is represented by many variables with , equality constraint and inequalities for all such that added to the optimization problem. Now, the integrality constraints can replace the original integrality constraint.
Binarized Flattening
An alternative to branch and bound consists in flattening the integrality constraints during the preprocessing stage, and moving the complexity to the logic manipulation abilities provided by the SMT solvers by using the OR operator. More precisely, this method can be imagined as a sort of “extreme” version of branch and bound, which tries to prevent all the integrality violations by adding specific constraints before starting the actual solving of the optimization problem. After that, a simple optimization without any branch and bound method can be started, thus the Integrality Management layer can be set to “Disabled”. This approach also assumes that the variables involved in integrality constraints are bounded.
Formally, for each integrality constraint where is bounded by we add the assertion
(1) 
If this approach is applied as it is, then it would be dramatically inefficient. In particular, for each variable involved in integrality constraints, it generates an assertion containing disjunctions where is , where and still represent the upper and the lower bound for . This means that if the lower bound of is and the upper bound is (realistic numbers for many problems), this adds an assertion with disjunctions. For hundreds of such integrality constraints, the size of the resulting new feasibility problem would be hard to manage.
But, if this approach is combined with binarization, then each variable involved in integrality constraints would be bounded just by 0 and 1, then only one disjunction is needed to exclude all the noninteger values between 0 and 1. In particular, if is an integrality constraint in a binarized problem, then adding the assertion
(2) 
is sufficient to “flatten” it. One may wonder whether the fact that such a simple assertion can be added to flatten a binarized program is counterbalanced by the fact that many variables may be needed to represent the original variables as sequences of bits.
For the above example, binarization needs only “binary” variables to represent it with . Since just one disjunction is used to flatten each “binary” variable, only disjunctions need to be added instead of if flattening is applied to the binary representation of .
To the best of our knowledge, binarized flattening is a novel approach and the experimental results in Section 5 clearly show that it has been the most valuable method implemented in the ManyOpt tool, as it has been fundamental to efficiently solve many problems which have not been solved within given timeouts by the methods based on branch and bound.
4.3 Integrality management: onebyone vs allinone
Most of the methods we implemented for integrality management are based on branch and bound techniques, but here we also move part of the combinatorial complexity to the lower layers.
The onebyone approach This approach is similar to a standard branch and bound approach. But there is an important difference: assume that a feasible solution is found in which at least one integrality constraint is violated. Then choose one variable whose value violates one of the integrality constraints and add the assertion to the optimization problem. Since disjunctions are supported in SMT as assertions, this is well defined and avoids a split into two optimization problems by delegating such combinatorial complexity to the SAT engine of the SMT solver.
The allinone approach We call the above approach onebyone, as each iteration only adds a sole assertion about a sole variable. But SMT solvers allow us to add more than one constraint at a time, leading to the allinone approach; it collects all variables whose values violate integrality constraints in a feasible solution and adds the above disjunction as an assertion but for all such variables simultaneously.
Delegating to lower layers As explained in section 3, it is also possible to disable integrality management. In this case, the whole complexity due to integrality constraints is moved directly to lower levels. Considering the current components of the optimization stack in ManyOpt, this would mean that such complexity is moved to the Feasibility Checking layer, where SMT solvers can provide support since the variables involved in integrality constraints can be represented using integer variables directly in SMT, and theories for mixed integer arithmetic are used by the SMT solver when it is called for feasibility checking.
4.4 Continuous relaxation optimization
As explained in section 3, this layer provides algorithms to solve NLP problems, and it relies on the feasibility checking layer.
The naive approach This method is quite simple but it can be effective in many circumstances, and it has been used for optimization. It consists of a loop in which a value is found for the objective function using the feasibility checker, and an attempt to find a lower value is made by adding an assertion saying that the objective function must be smaller than that value. When the problem will become infeasible, then the last found objective value, if there is one, is the optimal solution. See [18] for a more detailed description of this method.
Unbounded binary search We use here unbounded binary search as developed and used in [4] for optimization using an SMT solver. The unbounded binary search is composed of two main phases.

The bounds search, in which initial lower and upper bounds are found such that the optimal value of the objective is between such bounds.

The bisection phase, which is the actual binary search, where the interval between the lower bound and the upper bound is restricted by splitting it in two equal parts, until the optimal value of the objective is found, relative to the specified accuracy.
For more details about this method, see [4].
Hybrid method Our novel Hybrid method exploits the fact that Naive and Unbounded Binary Search methods can be written such that they have a similar structure and they share the same invariants. It is still composed of two phases like unbounded binary search, where first upper and lower bounds are determined for the objective, then the interval is restricted to find the optimal value. But in each phase, “naive steps” are used to determine whether an interval is empty in order to stop the algorithm if the current solution is already the optimal one.
Intuitively, the performance of this method on a specific NLP problem is expected to be closer to the performance of the best method (naive or unbounded binary search) on the same problem. This is confirmed by almost all the experiments in Section 5.
4.5 Feasibility checking: SMT solving
Originally, the ManyOpt tool was relying exclusively on Z3 as a feasibility checker, since Z3 provides a welldocumented Python API which has been used by our algorithms. But later, thanks to the PySMT framework [23], we have refactored the project and we made it independent of the underlying SMT solver, by creating a Feasibility Checking layer in which several solvers are available. More precisely, by using PySMT we now also support the SMT solvers MathSAT [10], CVC4 [2] and YICES [17].
Among the abovementioned solvers, Z3 is still the only one which has a relevant support for nonlinear arithmetics, especially the nonmixed one, i.e. with real arithmetics only. The other solvers provide a good support for linear arithmetics and can be used for MILP problems. Besides, they implicitly provide some support for MINLP problems where nonlinearity is due to particular combinations of linear functions, e.g. absolute values.
But, courtesy to the PySMT library, the ManyOpt tool also supports any external SMTLIB 2.0 compliant solver through POSIX piping by just providing the solver executable file. Therefore, ManyOpt can be used with SMT solvers specifically designed to reason about nonlinear real arithmetics, like [9], [21], [22]. Such solvers have not been tested yet, but it will be interesting to use them in future work as feasibility checker with a parallel combination of our techniques for Integrality Management and Continuous Relaxation Optimization.
5 Experimental results and validation
Here we discuss the salient experimental results which have been obtained using the ManyOpt tool. The main source of benchmarks for our experiments is MINLPLIB 2.0 [11], which contains a collection of more than one thousand MINLP and NLP problems in many formats, including the OSiL format we are able to parse in our tool. We also tested some MILP benchmarks available in MPS format from the MIPLIB2010 portal [28], and we also ran the tool ManyOpt with some complex NLP benchmarks from the Chemical Engineering world. All the findings are reported below.
5.1 Overall evaluation for ManyOpt
We started the experimentation with ManyOpt by taking the benchmarks from MINLPLIB and by starting to run all of them sorted by the benchmark size (smallest first), in order to be able to start with a reasonable timeout (30 minutes) for ManyOpt to get the solutions. The experimentation is still in progress, and if needed we will also increase the timeout as the benchmarks will increase in size and complexity. In this case, the “size” of a benchmark has been chosen to be the OSiL file size, in order to consider at the same time the number of variables, the number of constraints and the complexity of constraints.
So far we have ran all the first 193 experiments in the MINLPLIB archive sorted by size, not including the ones which have been filtered out because they were containing transcendental functions. The exponential functions are the only nonpolynomial functions accepted by our tool, but usually they are not solved because the SMT solving tools have almost no support for exponential functions. But we included them in the experimentation.
The machine used for the experiments is a 12core Intel®Xeon®E52640 2.50 GHz CPU with 47GB of RAM memory. Of the 193 MINLPLIB experiments ran so far, 56 were MINLP and 138 were NLP. For a fair analysis, we split up the experimental session into two separate ones in order to avoid running one category more than the other one (e.g. NLP more than MINLP). All the complete logs from the experiments are available in the ManyOpt tool webpage [33], , where for each experiment many details are logged depending on the chosen features, e.g. the bisection time for unbounded binary search, or the number of branch and bound iterations.
For the 193 experiments we ran, excluding only one MINLPLIB benchmark, the ManyOpt always returned the right solution when not exceeding the timeout. Clearly, the solution provided by ManyOpt was right with respect to the given accuracy, which has been set to .
We ran our experiments using 18 feature vectors in parallel for MINLP processes, by choosing all the possible methods in each layer, except for the feasibility layer, where we just chose Z3 because it can support nonlinear arithmetic.
For the NLP problems, preprocessing and integrality management features are not applicable because there are no integrality constraints. Thus, ManyOpt runs only 3 feature vectors in parallel for those problems.
As said above, the logs produced by the experiments are all available in the ManyOpt tool webpage [33]. Below we summarize the most important facts conveyed by the experimentation.

Regarding MINLP problems, our tool found the right solution within the given timeout, in about 68% of the problems.

Our tool solved about 63% of the NLP problems within the given timeout.
These are very high percentages, since the underlying Z3 solver used for the experimentation has low support for nonlinear and mixed arithmetic. Plus the percentage is surely higher if we also filtered out some problems containing exponential expressions; we will soon find a way to automatically filter them out and the updated results will be shown on the tool website.
Observation Compared to the current literature, this is an important step forward in supporting nonlinear and mixed constraints in SMTbased optimization. The complementarity of our methods was crucial to obtain this; we analyze these methods experimentally layer by layer as discussed below.
We also tried to solve some complex and widely believed to be challenging problems from Chemical Engineering, still with a 30 minutes timeout, and we solved 4 different problems in [14, 6, 24]. In particular, [14] reports that a popular nonlinear solver like MINOS [35] provided a poor solution for the example 2 in that paper, while our tool found the right solution within the timeout.
5.2 Layer analysis: preprocessing and integrality management
benchmark  allinone  onebyone  nobb  bin_allinone  bin_onebyone  bin_flattening 

hmittelman  ∎  ∎  ∎  ∎  ∎  1.08 
nvs04  0.38  0.46  ∎  ∎  ∎  
nvs06  0.81  0.39  ∎  ∎  1.08  
nvs07  0.21  5.59  12.22  ∎  0.19  
nvs10  0.23  ∎  ∎  149.79  ∎  0.06 
nvs16  0.27  0.27  ∎  ∎  ∎  
gear  0.28  0.33  2.11  4.45  112.07  
prob02  ∎  ∎  ∎  ∎  15.54  
prob03  ∎  ∎  ∎  ∎  0.02  
st_miqp3  ∎  ∎  0.03  ∎  ∎  ∎ 
nvs15  2.85  ∎  ∎  ∎  0.03  
st_testph4  ∎  ∎  ∎  ∎  0.06  
st_miqp2  0.24  0.49  ∎  ∎  15.28  
st_miqp1  ∎  ∎  ∎  ∎  0.05  
st_test1  0.17  ∎  3.11  ∎  0.03  
st_test2  ∎  ∎  ∎  ∎  ∎  16.38 
ex1221  0.48  1.17  ∎  0.65  2.45  0.19 
fac1  43.58  120.12  ∎  ∎  14.70  
st_e13  14.09  14.11  22.12  41.21  0.02  
sporttournament06  ∎  ∎  ∎  ∎  4.75  
gbd  0.03  0.03  0.02  0.04  0.04  0.02 
st_e27  ∎  ∎  ∎  ∎  ∎  0.16 
alan  0.11  0.23  ∎  0.14  0.55  0.09 
nvs01  1.55  2.55  ∎  ∎  ∎  
gear4  ∎  ∎  19.12  91.31  15.92  
tln2  ∎  10.32  0.90  16.64  0.07  
ex1263a  ∎  ∎  ∎  ∎  1.16 
After analyzing the overall experimental results for ManyOpt, here we show some experimental data in which single feature vectors have been ran instead of the whole ManyOpt tool, in order to show the performances of each feature in the ManyOpt layers. Because of time constraints, since such single feature vectors have been executed one by one sequentially, for these experiments the timeout has been reduced to 150 seconds (otherwise, since 18 feature vectors have been tested, each single experiment could have taken up to 9 hours).
For some representative benchmarks, Figure 2 compares the performance of all the methods from the Preprocessing and Integrality Management layers, since both layers manipulate integrality constraints. Each column represents a combination of methods for these two layers. More precisely, the columns show nonbinarized allinone and onebyone methods, the disabled integrality management (represented with nobb, i.e. no branch and bound), which just delegates the integrality management to the SMT solver, the binarized allinone and onebyone methods and finally the binarized flattening method. For each combination, the best result obtained when combining it with the three methods for continuous relaxation optimization is reported.
The little white squares in the paper represent a case in which the corresponding combination of features did not manage to find the solution within the timeout. In the nobb column (i.e. disabled integrality management), a black square means that the SMT solver returned because of some incomplete theory related to mixed and nonlinear arithmetic.
The benchmarks reported here have been chosen to show behaviors which occur in general in the experiments made so far, and the main observations to make based on these results are the following:
The first thing to observe is the importance of the binarized flattening method. The figure shows many cases (see e.g. hmittelman, prob02, prob03, st_testph4, and others) in which binarized flattening has been the only method which managed to obtain the optimum efficiently, while all the other ones exceeded the timeout. Binarized flattening has been a crucial method in order to solve most of the problems even if Z3 has low support for nonlinear and mixed arithmetic.
Looking at the table, we notice that the nonbinarized allinone approach also has an important role in solving. In many cases, it performs better than binarized flattening, or it solves the problem while binarized flattening exceeds the timeout. Together with binarized flattening, the nonbinarized allinone mode is fundamental for finding the solution for an important fraction of the MINLP problems tested so far. The complementarity relationship between these two approaches is clear (one is binarized, the other one is nonbinarized, one applies branch and bound dynamically, the other one statically flattens all the integrality conditions a priori, etc.), and their parallel execution lets us get the benefits from both of them, depending on the type of optimization problem which must be solved.
The other features shown in the table can also win occasionally. For example, the nobb approach, which moves integrality management towards SMT solving, has been the only method which found a solution for the st_miqp3 benchmark. And there are other cases in which the onebyone approach may perform better than allinone (see nvs06) and cases in which binarized branch and bound performs better than the nonbinarized one (see tln2, where the binarized allinone approach solves the problem in 0.9 seconds and the nonbinarized allinone approach exceeds the timeout).
Observation Our experimental results for ManyOpt show clear benefits of its approach, as seen in Fig. 2 and Fig. 3. If we had delegated mixed arithmetics to the SMT solver, e.g., then we would have solved almost no MINLP problems instead of 68% as there is limited support for the combination of mixed and nonlinear arithmetics by SMT solvers. This is also clear from the many black squares (i.e. SMT solver returning ) in the nobb column in the table, where the mixed nonlinear complexity handling is delegated to the SMT solving.
5.3 Layer by layer analysis: continuous relaxation optimization
benchmark  U.B.S  Naive  Hybrid 

st_e19  0.19  1.02  0.34 
chance  0.26  1.63  0.53 
linear  8.24  4.67  4.91 
circle  3.47  4.92  4.82 
pointpack02  0.19  206.98  0.47 
house  1.03  ∎  2.04 
haverly  0.61  ∎  1.24 
wastewater02m1  ∎  308.82  ∎ 
dispatch  ∎  38.66  ∎ 
st_e22  0.15  1.48  0.27 
sambal  219.34  ∎  527.86 
immun  24.17  ∎  47.19 
Figure 3 shows the performance of the three features available in the Continuous Relaxation Optimization layer, namely the unbounded binary search (U.B.S column), the naive method and our hybrid method.
Since the Continuous Relaxation Optimization features can also be used in NLP problems, here we show the results of some tests with NLP benchmarks, where the timeout is 10 minutes and only three features vectors have been tested for each benchmark.
The benchmarks in this table have been chosen to show two important facts: the first one is the complementarity between the Unbounded Binary Search and the Naive method. In particular, while the naive search technique, has been shown to often work well with SATbased optimization (see MiniSAT+), the unbounded binary search technique is faster in scanning big intervals containing many valid objective values. The table shows how big the gaps between the two techniques can be. For example, one of the two methods takes just a few seconds, while the other one even exceeds the 10 minutes timeout (see the house, haverly, wastewater02m1 and dispatch benchmarks for example).
The second important fact shown in the table is the benefit of the hybrid feature in mitigating the gap between unbounded binary search and naive search, as the hybrid feature combines them by exploiting the common invariants of their algorithms. As the table shows, its performance is almost always (but with some exceptions) closer to the best feature between U.B.S and Naive, and according to the overall experiments for ManyOpt it also wins very often (in about 20% of cases, see the logs in the ManyOpt tool webpage [33] for details).
6 Related work
We already discussed related work in the area of SMT solving and will therefore focus here on related work in optimization. Optimization problems with logical formulae frequently arise in practice; examples in process systems engineering include: choosing between multiple possible treatment technologies in a waste water treatment plant [27] and designing distillation column configurations [12]. Typically used techniques in process engineering will first translate logical rules into mathematical constraints (e.g., as detailed in [19]) and then solve the resulting MINLP problem; this process of formulating a mathematical optimization problem based on logical constraints and then transforming the resulting problem into a MINLP problem is known as Generalized Disjunctive Programming (GDP) [30]. Our proposed method of solving MINLP using SMT technology is therefore complementary to GDP; GDP is an excellent formulation technology, but transforming to a MINLP problem and then using traditional optimization methods eliminates the possibility of directly exploiting logical constraints in the original formulation.
With respect to MINLP solvers based on traditional branchandbound approaches [1, 5, 32, 34, 39], the advantage of the overall approach taken in tool ManyOpt is that – as in most SMTbased approaches – there is support for incremental strategies such as push/pop. Typical MINLP solvers will fathom, i.e. discard, regions of the tree as soon as they determine that this region cannot include the global solution.
7 Conclusions and Future Work
We have presented ManyOpt, an extensible tool capable of solving nonlinear and mixed optimization problems by relying on SMT solving, thanks to the parallel execution of complementary reduction techniques. While some of those reductions are inspired by existing techniques, other reductions we developed and integrated into ManyOpt – such as binarized flattening – are novel and crucial for leveraging the optimization capabilities of our tool. The experimental results confirm the effectiveness of our tool in solving MINLP problems, which are not solvable using other stateoftheart SMTbased optimizers, as these have almost no support for mixed nonlinear arithmetic.
The tool’s architecture is presented in layers that contain a number of features, and the tool is extensible by adding more features into such layers for further experimentation. In future work, we mean to extend ManyOpt in this manner and test these new features and their combinations with further experiments, for example by including the SMT optimizer Z in the continuous relaxation optimization layer. This is interesting as this would not require Z to deal with mixed arithmetic but only with nonlinear arithmetic.
We also would like to implement the capability of dynamic parallel execution in which, instead of statically determining feature vectors a priori, the fastest features are chosen dynamically during the execution. For example, while executing the allinone method, the program may run the naive, unbounded binary search and hybrid methods, and it may then obtain the solution by a different method in each allinone iteration.
Another important line of investigation is to understand how features used in parallel executions can share information to use such learned insights for improving the precision and performance of ManyOpt executions.
Our tool can easily incorporate external constraints to a MINLP problem, as such problems are represented in SMTLIB 2.0. We will collaborate with Chemical Process Engineers to explore the benefits of such enrichments of MINLP problems, in terms of decreasing running times, decreasing the frequency of replies, and enriching the problem specification so that “whatif” questions can be answered with supporting witness information.
We would like to exploit the POSIX piping within ManyOpt to support other external SMT solvers as feasibility checkers, notably some SMT solvers specifically developed for some subclasses of nonlinear real arithmetics [9, 21, 22]. Although such solvers are currently designed for real arithmetics only, they can be used in our tool, since we would use them to check the feasibility of continuous relaxations of MINLP problems.
Finally, we would like to understand how search methods from Artificial Intelligence, for example Conflictdirected A* [42], could interact with – or be integrated in – our tool and the approaches presented in this paper.
Openaccess Code and Research Data: The code for tool ManyOpt and more information on our experimental data and results are available at
Acknowledgments: This work was supported by the UK EPSRC with Fees Award and grants EP/N023242/1 and EP/N020030/1. We thank Ruth Misener and Miten Mistry for discussions and advice on MINLP and the use of SMT in solving such problems. Suleeporn Sujichantararat wrote a MSc thesis [41], supervised by the second author, that shadowed some aspects of the research program for the tool reported in this paper.
References
 [1] T Achterberg, T Berthold, T Koch, and K Wolter. Constraint integer programming: a new approach to integrate CP and MIP. In Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, 2008.
 [2] Clark Barrett, Christopher L Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. CVC4. In International Conference on Computer Aided Verification, pages 171–177. Springer, 2011.
 [3] M S Bazaraa, H D Sherali, and C M Shetty. Nonlinear programming: theory and algorithms. John Wiley & Sons, 2013.
 [4] P Beaumont, N Evans, M R A Huth, and T Plant. Confidence analysis for nuclear arms control: SMT abstractions of Bayesian Belief Networks. In Computer Security–ESORICS 2015, pages 521–540. Springer, 2015.
 [5] P Belotti, J Lee, L Liberti, F Margot, and A Wächter. Branching and bounds tightening techniques for nonconvex MINLP. Optim Method Softw, 24(45):597–634, 2009.
 [6] Aharon BenTal, Gideon Eiger, and Vladimir Gershovitz. Global minimization by reducing the duality gap. Mathematical programming, 63(13):193–212, 1994.
 [7] N Bjørner and AD Phan. Zmaximal satisfaction with Z3. In Proceedings of International Symposium on Symbolic Computation in Software Science (SCSS), 2014.
 [8] HJ K H Boehm and R Cartwright. Exact real arithmetic: Formulating real numbers as functions. Rice University, Department of Computer Science, 1988.
 [9] Cristina Borralleras, Salvador Lucas, Rafael NavarroMarset, Enric RodríguezCarbonell, and Albert Rubio. Solving nonlinear polynomial arithmetic via sat modulo linear arithmetic. In International Conference on Automated Deduction, pages 294–305. Springer, 2009.
 [10] Roberto Bruttomesso, Alessandro Cimatti, Anders Franzén, Alberto Griggio, and Roberto Sebastiani. The MathSAT4 SMT solver. In International Conference on Computer Aided Verification, pages 299–303. Springer, 2008.
 [11] M R Bussieck, A S Drud, and A Meeraus. MINLPLib  a collection of test models for mixedinteger nonlinear programming. INFORMS J Comput., 15(1):114–119, 2003.
 [12] J. A. Caballero and I. E. Grossmann. Generalized disjunctive programming model for the optimal synthesis of thermally linked distillation columns. Ind. Eng. Chem. Res., 40(10):2260–2274, 2001.
 [13] R Carvajal, S Ahmed, G Nemhauser, K Furman, V Goel, and Y Shao. Using diversification, communication and parallelism to solve mixedinteger linear programs. Operations Research Letters, 42(2):186–189, 2014.
 [14] Pedro M Castro, Henrique A Matos, and Augusto Q Novais. An efficient heuristic procedure for the optimal design of wastewater treatment systems. Resources, conservation and recycling, 50(2):158–185, 2007.
 [15] L De Moura and N Bjørner. Z3: An efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340. Springer, 2008.
 [16] L De Moura and N Bjørner. Satisfiability modulo theories: introduction and applications. Communications of the ACM, 54(9):69–77, 2011.
 [17] Bruno Dutertre and Leonardo De Moura. The YICES SMT solver. Tool paper at http://yices. csl. sri. com/toolpaper. pdf, 2(2), 2006.
 [18] N Eén and N Sörensson. Translating pseudoboolean constraints into sat. Journal on Satisfiability, Boolean Modeling and Computation, 2:1–26, 2006.
 [19] C A Floudas. Nonlinear and mixedinteger optimization: fundamentals and applications. Oxford University Press, New York, NY, 1995.
 [20] Robert Fourer, Jun Ma, and Kipp Martin. Osil: An instance language for optimization. Computational optimization and applications, 45(1):181–203, 2010.
 [21] Martin Franzle, Christian Herde, Tino Teige, Stefan Ratschan, and Tobias Schubert. Efficient solving of large nonlinear arithmetic constraint systems with complex boolean structure. Journal on Satisfiability, Boolean Modeling and Computation, 1:209–236, 2007.
 [22] Sicun Gao, Soonho Kong, and Edmund M Clarke. dReal: An SMT solver for nonlinear theories over the reals. In International Conference on Automated Deduction, pages 208–214. Springer, 2013.
 [23] Marco Gario and Andrea Micheli. PySMT: a solveragnostic library for fast prototyping of smtbased algorithms. In Proceedings of the 13th International Workshop on Satisfiability Modulo Theories (SMT), pages 373–384, 2015.
 [24] Co A Haverly. Studies of the behavior of recursion for the pooling problem. ACM Sigmap Bulletin, (25):19–28, 1978.
 [25] R C Jeroslow. There cannot be any algorithm for integer programming with quadratic constraints. Operations Research, 21(1):221–224, 1973.
 [26] R Kannan and C L Monma. On the computational complexity of integer programming problems. Springer, 1978.
 [27] R. Karuppiah and I. E. Grossmann. Global optimization for the synthesis of integrated water systems in chemical processes. Comput. Chem. Eng., 30:650 – 673, 2006.
 [28] Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E Bixby, Emilie Danna, Gerald Gamrath, Ambros M Gleixner, Stefan Heinz, et al. Miplib 2010. Mathematical Programming Computation, 3(2):103–163, 2011.
 [29] D Kroening and O Strichman. Decision procedures: an algorithmic point of view. Springer Science & Business Media, 2008.
 [30] S. Lee and I. E. Grossmann. A global optimization algorithm for nonconvex generalized disjunctive programming and applications to process systems. Comput. Chem. Eng., 25(1112):1675 – 1697, 2001.
 [31] Yi Li, Aws Albarghouthi, Zachary Kincaid, Arie Gurfinkel, and Marsha Chechik. Symbolic optimization with SMT solvers. ACM SIGPLAN Notices, 49(1):607–618, 2014.
 [32] Y Lin and L Schrage. The global solver in the LINDO API. Optim Method Softw, 24(45):657–668, 2009.
 [33] Manyopt tool website. https://bitbucket.org/andreacalliadiddio/manyopt
 [34] R Misener and C A Floudas. ANTIGONE: algorithms for continuous/integer global optimization of nonlinear equations. J Glob Optim, pages 1–24, 2014.
 [35] Bruce A Murtagh and Michael A Saunders. Minos 5.0 user’s guide. Technical report, DTIC Document, 1983.
 [36] K G Murty and S N Kabadi. Some NPcomplete problems in quadratic and nonlinear programming. Mathematical programming, 39(2):117–129, 1987.
 [37] Optimization services website. https://projects.coinor.org/OS
 [38] OSiL XML schema. http://www.optimizationservices.org/schemas/OSiL.xsd
 [39] N. V. Sahinidis. BARON: A general purpose global optimization software package. J Glob Optim, 8(2):201–205, 1996.
 [40] R Sebastiani and S Tomasi. Optimization modulo theories with linear rational costs. ACM Transactions on Computational Logic (TOCL), 16(2):12, 2015.
 [41] Suleeporn Sujichantararat. Optimization meets Constraint Solving: MINLP meets SMT. MSc Thesis for the MSc Degree in Computing of Imperial College London, September 2015.
 [42] Brian C. Williams and Robert J. Ragno. Conflictdirected A* and its role in modelbased embedded systems. Discrete Applied Mathematics 155(12): 1562 – 1595 (2007).