A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks
Abstract
Verification of neural networks enables us to gauge their robustness against adversarial attacks. Verification algorithms fall into two categories: exact verifiers that run in exponential time and relaxed verifiers that are efficient but incomplete. In this paper, we unify all existing LPrelaxed verifiers, to the best of our knowledge, under a general convex relaxation framework. This framework works for neural networks with diverse architectures and nonlinearities and covers both primal and dual views of robust verification. We further prove strong duality between the primal and dual problems under very mild conditions. Next, we perform largescale experiments, amounting to more than 22 CPUyears, to obtain exact solution to the convexrelaxed problem that is optimal within our framework for ReLU networks. We find the exact solution does not significantly improve upon the gap between PGD and existing relaxed verifiers for various networks trained normally or robustly on MNIST and CIFAR datasets. Our results suggest there is an inherent barrier to tight verification for the large class of methods captured by our framework. We discuss possible causes of this barrier and potential future directions for bypassing it.
A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks
Hadi Salman^{†}^{†}thanks: Work done as part of the Microsoft AI Residency Program. Microsoft Research AI hadi.salman@microsoft.com Greg Yang Microsoft Research AI gregyang@microsoft.com Huan Zhang UCLA huan@huanzhang.com ChoJui Hsieh UCLA chohsieh@cs.ucla.edu Pengchuan Zhang Microsoft Research AI penzhan@microsoft.com
noticebox[b]Preprint. Work in progress.\end@float
1 Introduction
Neural networks (NNs) have had a lot of successes in tasks such as image classification (Krizhevsky et al., 2012; He et al., 2016) and speech recognition (Hinton et al., 2012). However, it has been shown that NNs are vulnerable to adversarial examples, for instance perturbed images which are visually indistinguishable from the original to a human, but which can change the NN’s classification dramatically (Szegedy et al., 2013; Goodfellow et al., 2015). This becomes a serious problem when NNs are applied to safetycritical applications, or those where security is a requirement. Such concern has prompted very active research in adversarial attack (designing adversarial examples) (Goodfellow et al., 2015; Carlini & Wagner, 2017; Chen et al., 2017), defense (train or inferencetime procedures to nullify effectiveness of adversarial examples) (Madry et al., 2017; Wong & Kolter, 2018; Raghunathan et al., 2018a), and verification (certifying that no small perturbations of a given input can cause the NN to change its prediction) (Katz et al., 2017; Dvijotham et al., 2018a; Weng et al., 2018). This paper is concerned with the last, robust verification.
Many recent works have proposed robust verification methods (Katz et al., 2017; Ehlers, 2017; Hein & Andriushchenko, 2017). Typically, such algorithms try to exactly find or upper bound an adversarial loss, defined to be the maximum gap between the network’s correct logit and any of its incorrect logits, over all possible normbounded perturbations of the input (see eq. 1). Alternatively, they exactly find or lower bound the minimum adversarial distortion , defined to be the radius of the largest ball (in a specified norm), around the input, that contains no adversarial examples. The algorithms devised so far can be roughly categorized as exact (Ehlers, 2017; Katz et al., 2017; Tjeng et al., 2019b) or relaxed methods (Wong & Kolter, 2018; Zhang et al., 2018; Singh et al., 2018), and among the latter, the most dominant category is convex relaxation, which tries to relax the exact adversarial optimization problem into a convex problem. Our first main contribution is to give a unifying treatment of convex relaxation used in a large number of recent algorithms (Fig. 1) and reveal the relationships between them. We further prove that these relaxed problems satisfy strong duality under very mild conditions, connecting relaxed verifiers in the primal space (Zhang et al., 2018; Weng et al., 2018) to those in the dual space (Dvijotham et al., 2018c; Wong & Kolter, 2018).
For any NN, our framework prescribes a unique tightest relaxation, whose optimal value presents a barrier for any method described by the framework. How close is to the true adversarial loss? In our second main contribution, we perform extensive experiments with deep ReLU networks to compute and compare with the LPrelaxed dual formulation of Wong & Kolter (2018) (which, as our framework will show, approximates ), the PGD attack of Madry et al. (2017), and the mixed integer linear programming (MILP) exact verifier of Tjeng et al. (2019b). Over different models, sizes, training methods, and datasets (MNIST and CIFAR10), we find that our tightest relaxation only very slightly improves the lower bound of the minimum adversarial distortion found by Wong & Kolter (2018), especially when compared with the upper bound provided by the PGD attack, which is consistently 1.5 to 5 times larger. For networks trained using the LPrelaxed dual formulation of Wong & Kolter (2018), when is large and the robust error is also large, our tighest relaxation improves the certified robust error significantly in relative terms, but at most 7% in absolute terms. Furthermore, on both normally and PGDtrained MNIST classifiers, our tightest relaxation does not significantly close the gap in the robust error between the PGD lower bound or MILP exact answer on one hand, and the upper bound of Wong & Kolter (2018) on the other. Thus the convex relaxation barrier we discover theoretically indeed presents a significant challenge to existing algorithms empirically.
2 Preliminaries and Related Work
Robustness to adversarial attacks.
Consider a multiclass classification network , where is the logit for the th class. Given a nominal input , its nominal prediction , and a neighborhood , the following optimization problem determines if the network is robust to adversarial attacks within this neighborhood:
(1) 
If the minimal value is positive for all , the network is robust at this point within . We refer the reader to Dvijotham et al. (2018c) and Qin et al. (2019) for other robust verification problems which can be solved with similar methods.
Exact verifiers and NPcompleteness.
For ReLU networks (piecewise linear networks in general), exact verifiers solve the robust verification problem (1) by typically employing MILP solvers (Cheng et al., 2017; Lomuscio & Maganti, 2017; Dutta et al., 2018; Fischetti & Jo, 2017; Tjeng et al., 2019a; Xiao et al., 2019) or Satisfiability Modulo Theories (SMT) solvers (Scheibler et al., 2015; Katz et al., 2017; Carlini et al., 2017; Ehlers, 2017). However, due to the NPcompleteness for solving such a problem (Katz et al., 2017; Weng et al., 2018), it can be really challenging to scale these to large networks. It can take Reluplex (Katz et al., 2017) several hours to find the minimum distortion of an example for a ReLU network with 5 inputs, 5 outputs, and 300 neurons. Recent work by Tjeng et al. (2019b) uses MILP to exactly verify mediumsize networks, but the verification time is very sensitive to how a network is trained; for example, it is fast for networks trained using the LPrelaxed dual formulation of Wong & Kolter (2018), but much slower for normally trained networks. A concurrent work by Xiao et al. (2019) trains networks with the objective of speeding up the MILP verification problem, but this compromises on the performance of the network.
Relaxed and efficient verifiers.
These verifiers solve a relaxed, but more computationally efficient, version of (1), and have been proposed from different perspectives. From the primal view, one can relax the nonlinearity in (1) into linear inequality constraints. This perspective has been previously explored as in the framework of “abstract transformers” (Singh et al., 2018, 2019b, 2019a; Gehr et al., 2018; Mirman et al., 2018), via linear outer bounds of activation functions (Zhang et al., 2018; Weng et al., 2018; Wang et al., 2018b, a), or via interval bound propagation (Gowal et al., 2018). From the dual view, one can study the dual of the relaxed problem (Wong & Kolter, 2018; Wong et al., 2018) or study the dual of the original nonconvex verification problem (Dvijotham et al., 2018c, b; Qin et al., 2019). In this paper, we unify both views in a common convex relaxation framework for NN verification, clarifying their relationships (as summarized in Fig. 1).
Raghunathan et al. (2018b) formulates the verification of ReLU networks as a quadratic programming problem and then relaxes and solves this problem with a semidefinite programming (SDP) solver. While our framework does not cover this SDP relaxation, it is not clear to us how to extend the SDP relaxed verifier to general nonlinearities, for example max pooling, something which can be done in our framework on the other hand. Other verifiers have been proposed to certify via an intermediary step of bounding the local Lipschitz constant (Hein & Andriushchenko, 2017; Weng et al., 2018; Raghunathan et al., 2018a; Zhang et al., 2019). These are outside the scope of our framework.
Combining exact and relaxed verifiers, hybrid methods have shown some effectiveness (Bunel et al., 2018; Singh et al., 2019a). In fact, many exact verifiers also use relaxation as a subroutine to speed things up, and hence can be viewed as hybrid methods as well. In this paper, we are not concerned with such techniques but only focus on relaxed verifiers.
3 Convex Relaxation from the Primal View
Problem setting.
In this paper, we assume that the neighborhood is a convex set. An example of this is , which is the constraint on in the adversarial attack model. We also assume that is an layer feedforward NN. For notational simplicity, we denote by and by . We define as,
(2)  
where , , is the input, and are the weight matrix and bias vector of the linear layer, and is a (nonlinear) activation function like (leaky)ReLU, the sigmoid family (including sigmoid, arctan, hyperbolic tangent, etc), and the pooling family (MaxPool, AvgPool, etc). Our results can be easily extended to networks with convolutional layers and skip connections as well, similar to what is done in Wong et al. (2018), as these can be seen as special forms of (2).
Consider the following optimization problem :
()  
s.t.  
where the optimization domain is
(3) 
If , , , and , then () is equivalent to problem (1). However, when we have better information about valid bounds and of , we can significantly narrow down the optimization domain and, as will be detailed shortly, achieve tighter solutions when we relax the nonlinearities. We denote the minimal value of by , or just when no confusion arises.
Obtaining lower and upper bounds by solving subproblems.
This can be done by recursively solving () with specific choices of and , which is a common technique used in many works (Wong & Kolter, 2018; Dvijotham et al., 2018c). For example, one can obtain , a lower bound of , by solving ; this shows that one can estimate and inductively in . However, we may have millions of subproblems to solve because practical networks can have millions of neurons. Therefore, it is crucial to have efficient algorithms to solve problem ().
General nonlinear specifications
are also of interest beyond the linear specification considered in (). These can be formulated as a special case of () by adding a nonlinear layer on top of the original network, and setting and . Because our framework can deal with general nonlinear layers, all the theoretical results in this paper can be applied to verification problems with nonlinear specifications as well.
Convex relaxation in the primal space.
Due to the nonlinear activation functions , the feasible set of () is nonconvex, which leads to the NPcompleteness of the robust verification problem (Katz et al., 2017; Weng et al., 2018). One natural idea is to do convex relaxation of its feasible set. Specifically, one can relax the nonconvex equality constraint to convex inequality constraints, i.e.,
()  
s.t.  
where ( ) is convex (concave) and satisfies for . We denote the feasible set of () by and its minimum by . Naturally, we have that is convex and .
For example, Ehlers (2017) proposed the following relaxation for the ReLU function :
(4) 
and the following for MaxPool :
(5) 
The optimal convex relaxation.
As a special case, we consider the optimal convex relaxation, where is the greatest convex function majored by , and is the smallest concave function majoring . They can be equivalently defined as
(6)  
(7) 
In Fig. 2, we show the optimal convex relaxation for several common activation functions. It is easy to see that (4) is the optimal convex relaxation for ReLU, but (5) is not optimal for the MaxPool function.
When is a nonlinear layer that has a vector output , the optimal convex relaxation may not have a simple analytic form as . Fortunately, if there is no interaction (as defined below) among the output neurons, the optimal convex relaxation can be given as a simple analytic form.
Definition 3.1 (noninteractive layer).
Let and be a nonlinear layer with input and output . For each output , let be the minimal set of ’s entries that affect , where . We call the layer noninteractive if the sets () are mutually disjoint.
Commonly used nonlinear activation layers are all noninteractive. It is obvious that all entrywise nonlinear layers, such as (leaky)ReLU and sigmoid, are noninteractive. A MaxPool layer with nonoverlapping regions (stride no smaller than kernel size) is also noninteractive. Finally, any layer with scalarvalued output is noninteractive. When we treat a general nonlinear specification (as proposed in Qin et al. (2019)) as an additional nonlinear layer , this layer is automatically noninteractive. This nice property ensures that our framework can deal with very general specifications.
The optimal convex relaxation of a noninteractive layer has a simple analytic form as below.
Proposition 3.2.
Convex relaxations not included in Problem ().
We emphasize that by optimal, we mean the optimal convex relaxation of the single nonlinear constraint (see Proposition (3.2)) instead of the optimal convex relaxation of the nonconvex feasible set of the original problem (). In fact, for neural networks with more than two hidden layers (), the optimal convex relaxation of the nonconvex feasible set of problem () is a strict subset of the feasible set of problem (), even with the tightest bounds and the optimal choice of and in (6) and (7). It is possible to obtain other (maybe tighter) convex relaxations, but it comes with more assumptions on the nonlinear layers and more complex convex constraints.
For example, Raghunathan et al. (2018b) rewrites the ReLU nonlinearity as a quadratic constraint, and then proposes a semidefinite programming (SDP) relaxation for the resulting quadratic optimization problem. Problem () does not cover this SDPrelaxation. Sometimes Problem () provides tighter relaxation than the SDPrelaxation, e.g., the case when there is only one neuron in a layer, while sometimes the SDPrelaxation provides tighter relaxation than Problem (), e.g., the examples provided in Raghunathan et al. (2018b). The SDPrelaxation currently only works for ReLU nonlinearity. It is not clear to us how to extend the SDPrelaxed verifier to general nonlinearities. On the other hand, Problem () can handle any noninteractive nonlinear layer and any nonlinear specification.
Greedily solving the primal with linear bounds.
Before resorting to duality for large problems, it is possible to solve () by overrelaxing the problem to efficiently give a lower bound for the objective. This can be done by defining exactly one linear upper bound and one linear lower bound for each activation function in () as follows,
Given these bounds for each activation, the primal variables can be greedily bounded in a layerbylayer manner: we can derive one linear upper and one linear lower bound of with respect to using the fact that , and that is linearly upper and lower bounded by and . Because a linear combination of linear bounds (coefficients are related to the entries in ) is a single linear bound, we can apply this technique again and replace with its upper and lower bounds with respect to , obtaining the bound for with respect to . Applying this repeatedly eventually leads to linear lower and upper bounds of with respect to the input .
This perspective covers AI (Gehr et al., 2018), FastLin (Weng et al., 2018), DeepZ (Singh et al., 2018) and Neurify (Wang et al., 2018b), where the proposed linear lower bound has the same slope as the upper bound, i.e., . The resulting shape is referred to as a zonotope in Gehr et al. (2018) and Singh et al. (2018). In CROWN (Zhang et al., 2018) and DeepPoly (Singh et al., 2019b), this restriction is lifted and they can achieve better verification results than FastLin and DeepZ. Fig. 1 summarizes the relationships between these algorithms. Importantly, each of these works has its own merits on solving the verification problem; our focus here is to give a unified view on how they overrelax in our framework. See Appendix A.2 for more discussions and other related algorithms.
4 Convex Relaxation from the Dual View
We now tackle the verification problem from the dual view and show its connection to the primal view.
Strong duality for the convex relaxed problem.
As in Wong & Kolter (2018), we introduce the dual variables for () and write its Lagrangian dual as
(8)  
Note that for any choice of , and , the value of the dual function lower bounds the optimal value of (), an thus we have
(9) 
This property is known as weak duality (Boyd & Vandenberghe, 2004). In the following, we prove that strong duality between () and (9) holds under some mild conditions.
Theorem 4.1.
If, for each , both and have a finite Lipschitz constant in the domain , then strong duality holds between () and (9), i.e.,
The assumption of the finite Lipschitz constant is so weak that one can always construct such and as long as is bounded in the domain . Notice that we do not make additional geometric assumptions (like Slater’s condition) in Theorem (4.1). Instead, we make use of geometric properties of NNs, i.e., their linear stability against perturbations in the layerwise computation, to obtain our strong duality results. We present the linear stability analysis in Lemma A.1 and provide the strong duality proof in Appendix A.3.
The optimal dual relaxation.
If we restrict ourselves to the dual of the convex relaxed problem (), the optimal (tightest) bound the dual problem can provide is exactly the bound the optimal convex relaxation ((6) and (7)) provides, thanks to the strong duality.
One approach to possibly obtain a tighter dual problem is to directly study the Lagrangian dual of the original problem (),
(10)  
which was first proposed in Dvijotham et al. (2018c). Note that for any choice of and , the value of the dual function lower bounds the optimal value of (). Thus,
(11) 
It would seem to be strictly better than (9). Unfortunately, the following theorem shows that the dual problem (11) can do no better than the dual of the convex relaxation (9).
Theorem 4.2.
Proof sketch.
First, we prove that for any , and , with , we have
(12) 
where and . Therefore, the dual problem (9) can be rewritten as an unconstrained optimization problem as
(13) 
The complete proof is in Appendix A.4. From (13) and (14), we can see that these two dual problems not only share the optimal value, but also share the same objective function! We would like to point out that Theorem 2 in Dvijotham et al. (2018c) is a special case of our Theorem 4.2, when applied to ReLU networks. Our proof makes use of the FenchelMoreau theorem (Theorem 12.2 in Rockafellar (2015)) to deal with general nonlinearities, which is different from that in Dvijotham et al. (2018c).
Theorem 4.2 combined with the strong duality result of Theorem 4.1 implies that the primal relaxation () and the two kinds of dual relaxations, (9) and (11), are all blocked by the same barrier — the bound that the optimal convex relaxation ((6) and (7)) provides.
Corollary 4.3.
Suppose that the activation functions () are noninteractive and are among the following functions: piecewise linear functions (e.g., ReLU and staircase functions^{1}^{1}1Note that we only require and defined in (6) and (7) be Lipschitz continuous, while itself may not be Lipschitz.), ELU, sigmoid, arctan, hyperbolic tangent, polynomials and common pooling functions (e.g., MaxPool and AvgPool). Assume that and are defined in (6) and (7), respectively. Then we have that the lower bound provided by the primal relaxation () and provided by the dual relaxation (11) are the same, i.e.,
We point out that the constraint on the activation function is to ensure the strong duality between () and (9) when and are defined in (6) and (7), respectively. This constraint is so weak that it is satisfied by nearly all commonly used activation functions, far more than the functions listed in Corollary 4.3.
Which problem to solve in practice?
Thanks to the strong duality, the same lower bound can be achieved from both the primal and the dual problems, and thus we have the freedom to choose the problem to solve. When the relaxed upper and lower bounds, i.e., and , are piecewise linear (e.g. (4) for ReLU networks), both the primal and dual problems are linear programs and can be efficiently solved by existing LP solvers (which is what we use in the coming sections). In other cases, we recommend to solve the dual problem (11) for two reasons. First, the primal relaxed problem () is a constrained optimization problem, and its constraints may not have a simple analytic form when and are not piecewise linear; see examples in Fig. 2. On the contrary, the dual problem (11) can be framed as an unconstrained optimization problem and its objective function has a simple analytic form for some common activation functions (Dvijotham et al., 2018c). Second, the optimization process of (11) can be stopped anytime to give a lower bound of , thanks to weak duality, but this is not true of the primal view. Of course, and must be in the form of (6) and (7) to achieve the optimal value.
Greedily solving the dual with linear bounds.
When the relaxed bounds and are linear, i.e.,
the dual objective (9) can be approximately solved (lower bounded) by a greedy algorithm. For }, this greedy algorithm fixes the already obtained suboptimal solution of , then computes a suboptimal solution for by maximizing the part of the objective function that only involves . We provide the algorithm details in Appendix A.5.
It turns out that this greedy algorithm exactly recovers the algorithm proposed in Wong & Kolter (2018) when
where represents the slope of the lower bound. When , the greedy algorithm also recovers FastLin (Weng et al., 2018), which explains the arrow from Wong & Kolter (2018) to Weng et al. (2018) in Fig. 1. When is chosen adaptively as in CROWN (Zhang et al., 2018), the greedy algorithm then recovers CROWN, which explains the arrow from Wong & Kolter (2018) to Zhang et al. (2018) in Fig. 1. See Appendix A.2 for more discussions on the relationship between the primal and dual greedy solvers.
5 Optimal LPrelaxed Verification
In the previous sections, we presented a framework that subsumes all existing convexrelaxed verification algorithms except that of Raghunathan et al. (2018b). For ReLU networks, being piecewise linear, these correspond exactly to the set of all existing LPrelaxed algorithms, as discussed above. We showed the existence of a barrier, , that limits all such algorithms. Is this just theoretical babbling or is this barrier actually problematic in practice?
In the next section, we perform extensive experiments on deep ReLU networks, evaluating the tightest convex relaxation afforded by our framework (denoted LPall) against the algorithm of Wong & Kolter (2018) (denoted LPgreedy) as well as another algorithm LPlast, intermediate in speed and accuracy between them. Both LPgreedy and LPlast solve the bounds by setting the dual variables heuristically (see previous section), but LPgreedy solves the adversarial loss in the same manner while LPlast solves this final LP exactly. We also compare them with the opposite bounds provided by PGD attack (Madry et al., 2017), as well as MILP exact results (Tjeng et al., 2019b).
For the rest of the main text, we are only concerned with ReLU networks, so () subject to (4) is in fact an LP.
5.1 LPall Implementation Details
In order to exactly solve the tightest LPrelaxed verification problem of a ReLU network, two steps are required: (A) obtaining the tightest preactivation upper and lower bounds of all the neurons in the NN, excluding those in the last layer, then (B) solving the LPrelaxed verification problem exactly for the last layer of the NN.
Step A: Obtaining Preactivation Bounds.
This can be done by solving subproblems of the orginial relaxed problem () subject to (4). Given a NN with layers, for each layer , we obtain a lower (resp. upper) bound (resp. ) of , for all neurons . We do this by setting
However, we need to solve an LP for each neuron, and practical networks can have millions of them. We utilize the fact that in each layer , computing the bounds and for each can proceed independently in parallel. Indeed, we design a scheduler to do so on a cluster with 1000 CPUnodes. See Appendix C for details.
Step B: Solving the LPrelaxed Problem for the Last Layer.
After obtaining the preactivation bounds on all neurons in the network using step (A), we solve the LP in () subject to (4) for all obtained by setting
Here, is the true label of the data point at which we are verifying the network. We can certify the network is robust around iff the solutions of all such LPs are positive, i.e. we cannot make the true class logit lower than any other logits. Again, note that these LPs are also independent of each other, so we can solve them in parallel.
Given any , LPall follows steps (A) then (B) to produce a certificate whether the network is robust around a given datapoint or not. LPlast on the other hand solves only step (B), and instead of doing step (A), it finds the preactivation bounds heuristically as in Wong & Kolter (2018).
6 Experiments
We conduct two experiments to assess the tightness of LPall: 1) finding certified bounds on the robust error of several NN classifiers, 2) finding certified bounds on the minimum adversarial distortion using different algorithms. All experiments are conducted on MNIST and/or CIFAR10 datasets.
Architectures.
We conduct experiments on a range of ReLUactivated feedforward networks. MLPA and MLPB refer to multilayer perceptrons: MLPA has 1 hidden layer with 500 neurons, and MLPB has 2 hidden layers with 100 neurons each. CNNsmall, CNNwidek, and CNNdeepk are the ConvNet architectures used in Wong et al. (2018). Full details are in Appendix B.1.
Training Modes.
We conduct experiments on networks trained with a regular crossentropy (CE) loss function and networks trained to be robust. These networks are identified by a prefix corresponding to the method used to train them: LPd when the LPrelaxed dual formulation of Wong & Kolter (2018) is used for robust training, Adv when adversarial examples generated using PGD are used for robust training, as in Madry et al. (2017), and Nor when the network is normally trained using the CE loss function. Training details are in Appendix B.2.
Experimental Setup.
6.1 Certified Bounds on the Robust Error
Table 1 presents the clean test errors and (upper and lower) bounds on the true robust errors for a range of classifiers trained with different procedures on MNIST. For both Adv and LPdtrained networks, the in Table 1 denotes the norm bound used for training and robust testing; for Normallytrained networks, is only used for the latter.
Lower bounds on the robust error are calculated by finding adversarial examples for inputs that are not robust. This is done by using PGD, a strong firstorder attack, or using MILP (Tjeng et al., 2019b). Upper bounds on the robust error are calculated by providing certificates of robustness for input that is robust. This is done using MILP, the dual formulation (LPgreedy) presented by Wong & Kolter (2018), or our LPall algorithm.
For the MILP results, we use the code accompanying the paper by Tjeng et al. (2019b). We run the code in parallel on a cluster with 1000 CPUnodes, and set the MILP solver’s time limit to 3600 seconds. Note that this time limit is reached for Adv and Nor, and therefore the upper and lower bounds are separated by a gap that is especially large for some of the Normally trained networks. On the other hand, for LPdtrained networks, the MILP solver finishes within the time limit, and thus the upper and lower bounds match.
For all Normally and Advtrained networks, we see that the certified upper bounds using LPgreedy and LPall are very loose when we compare the gap between them to the lower bounds found by PGD and MILP. As a sanity check, note that LPall gives a tighter bound than LPgreedy in each case, as one would expect. Yet this improvement is not significant enough to close the gap with the lower bounds.
This sanity check also passes for LPdtrained networks, where the LPgreedycertified robust error upper bound is, as expected, much closer to the true error (given by MILP here) than for other networks. For , the improvement of LPallcertified upper bound over LPgreedy is at most modest, and the PGD lower bound is tighter to the true error. For large , the improvement is much more significant in relative terms, but the absolute improvement is only . In this large regime, however, both the clean and robust errors are quite large, so the tightness of LPall is less useful.
6.2 Certified Bounds on the Minimum Adversarial Distortion
We are interested in searching for the minimum adversarial distortion , which is the radius of the largest ball in which no adversarial examples can be crafted. An upper bound on is calculated using PGD, and lower bounds are calculated using LPgreedy, LPlast, or our LPall, all via binary search. Since solving LPall is expensive, we find the bounds only for ten samples of the MNIST and CIFAR10 datasets. See Appendix E.1 for details.
Fig. 3 and 4 show the median percentage gap (defined in Appendix E.2) between the convexrelaxed algorithms and PGD estimates of for MNIST and CIFAR, respectively. Details are reported in Tables 2 and 3 in Appendix E.2.
On MNIST, the results show that for all networks trained Normally or via Adv, the certified lower bounds on are 1.5 to 5 times smaller than the upper bound found by PGD; for LPd trained networks, below 1.5 times smaller. On CIFAR10, the bounds are between 1.5 and 2 times smaller across all models.
The smaller gap for LPd is of course as expected following similar observations in prior work (Wong & Kolter, 2018; Tjeng et al., 2019b). Furthermore, the improvement of LPall and LPlast over LPgreedy is not significant enough to close the gap with the PGD upper bound. Note that similar results hold as well for randomly initialized networks (no training). To avoid clutter, we report these in Appendix F.
7 Discussion
In this work we first presented a convex relaxation framework that unifies all previous LPrelaxed verifiers. Then we performed extensive experiments to show that even the optimal convex relaxation for ReLU networks in this framework cannot obtain tight bounds on robust error in all cases we consider here. Thus any method will face a convex relaxation barrier as soon as it can be described by our framework.
Note that different applications have different requirements for the tightness of the verification, so our barrier could be a problem for some but not for others. In so far as the ultimate goal of robust verification is to construct a training method to lower certified error, this barrier is not necessarily problematic — some such method could still produce networks for which convex relaxation as described by our framework produces accurate robust error bounds. An example is the recent work of Gowal et al. (2018) which shows that interval bound propagation, which often leads to loose certification bounds, can still be used for verified training, and is able to achieve stateoftheart verified accuracy when carefully tuned. However, without a doubt, in all cases, tighter estimates should lead to better results, and we reveal a definitive ceiling on most current methods.
How to bypass the barrier?
The primal problem in our framework () has several possible sources of loosenes:

We relax the nonlinearity on a box domain . This relaxation is simple to perform, but might come at the cost of losing the correlation between the coordinates of and of obtaining a looser relaxation.

We solve for the bounds recursively, and we incur some gap for every recursion.

In the specific case of ReLU, we lose a bit every time we relax over an unstable neuron; one possible future direction is to combine branchandbound with convex relaxation to strategically split the domains of unstable neurons.
Any method that improves on any of the above issues can possibly bypass the barrier. On the other hand, exact verifiers (Katz et al., 2017; Ehlers, 2017), SDPbased verifiers (Raghunathan et al., 2018b, a), Lipschitzconstantbased verifiers (Zhang et al., 2019), and hybrid approaches (Bunel et al., 2018; Singh et al., 2019a) do not fall under the purview of our framework. In general, none of them are strictly better than the convex relaxation approach, sacrificing either speed or accuracy. However, it would be fruitful to consider combinations of these methods in the future, as done in Singh et al. (2019a).
We hope our work will foster much thought in the community toward new relaxation paradigms for tight robust verification.
References
 Boyd & Vandenberghe (2004) Boyd, S. and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
 Bunel et al. (2018) Bunel, R. R., Turkaslan, I., Torr, P., Kohli, P., and Mudigonda, P. K. A unified view of piecewise linear neural network verification. In Advances in Neural Information Processing Systems, pp. 4795–4804, 2018.
 Carlini & Wagner (2017) Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. IEEE, 2017.
 Carlini et al. (2017) Carlini, N., Katz, G., Barrett, C., and Dill, D. L. Provably minimallydistorted adversarial examples. arXiv preprint arXiv:1709.10207, 2017.
 Chen et al. (2017) Chen, P.Y., Zhang, H., Sharma, Y., Yi, J., and Hsieh, C.J. Zoo: Zeroth order optimization based blackbox attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15–26. ACM, 2017.
 Cheng et al. (2017) Cheng, C.H., Nührenberg, G., and Ruess, H. Maximum resilience of artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis, pp. 251–268. Springer, 2017.
 Diamond & Boyd (2016) Diamond, S. and Boyd, S. CVXPY: A Pythonembedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1–5, 2016.
 Domahidi et al. (2013) Domahidi, A., Chu, E., and Boyd, S. ECOS: An SOCP solver for embedded systems. In European Control Conference (ECC), pp. 3071–3076, 2013.
 Dutta et al. (2018) Dutta, S., Jha, S., Sankaranarayanan, S., and Tiwari, A. Output range analysis for deep feedforward neural networks. In NASA Formal Methods Symposium, pp. 121–138. Springer, 2018.
 Dvijotham et al. (2018a) Dvijotham, K., Garnelo, M., Fawzi, A., and Kohli, P. Verification of deep probabilistic models. CoRR, abs/1812.02795, 2018a. URL http://arxiv.org/abs/1812.02795.
 Dvijotham et al. (2018b) Dvijotham, K., Gowal, S., Stanforth, R., Arandjelovic, R., O’Donoghue, B., Uesato, J., and Kohli, P. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265, 2018b.
 Dvijotham et al. (2018c) Dvijotham, K., Stanforth, R., Gowal, S., Mann, T., and Kohli, P. A dual approach to scalable verification of deep networks. UAI, 2018c.
 Ehlers (2017) Ehlers, R. Formal verification of piecewise linear feedforward neural networks. In International Symposium on Automated Technology for Verification and Analysis, pp. 269–286. Springer, 2017.
 Fischetti & Jo (2017) Fischetti, M. and Jo, J. Deep neural networks as 01 mixed integer linear programs: A feasibility study. arXiv preprint arXiv:1712.06174, 2017.
 Gehr et al. (2018) Gehr, T., Mirman, M., DrachslerCohen, D., Tsankov, P., Chaudhuri, S., and Vechev, M. AI 2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP), 2018.
 Goodfellow et al. (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. ICLR, 2015.
 Gowal et al. (2018) Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715, 2018.
 He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
 Hein & Andriushchenko (2017) Hein, M. and Andriushchenko, M. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems (NIPS), pp. 2266–2276, 2017.
 Hinton et al. (2012) Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82–97, 2012.
 Katz et al. (2017) Katz, G., Barrett, C., Dill, D. L., Julian, K., and Kochenderfer, M. J. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pp. 97–117. Springer, 2017.
 Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
 Lomuscio & Maganti (2017) Lomuscio, A. and Maganti, L. An approach to reachability analysis for feedforward relu neural networks. arXiv preprint arXiv:1706.07351, 2017.
 Madry et al. (2017) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
 Mirman et al. (2018) Mirman, M., Gehr, T., and Vechev, M. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, pp. 3575–3583, 2018.
 Qin et al. (2019) Qin, C., Dvijotham, K. D., O’Donoghue, B., Bunel, R., Stanforth, R., Gowal, S., Uesato, J., Swirszcz, G., and Kohli, P. Verification of nonlinear specifications for neural networks. ICLR, 2019.
 Raghunathan et al. (2018a) Raghunathan, A., Steinhardt, J., and Liang, P. Certified defenses against adversarial examples. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1801.09344, 2018a.
 Raghunathan et al. (2018b) Raghunathan, A., Steinhardt, J., and Liang, P. S. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp. 10900–10910, 2018b.
 Rockafellar (2015) Rockafellar, R. T. Convex analysis. Princeton university press, 2015.
 Scheibler et al. (2015) Scheibler, K., Winterer, L., Wimmer, R., and Becker, B. Towards verification of artificial neural networks. In MBMV, pp. 30–40, 2015.
 Singh et al. (2018) Singh, G., Gehr, T., Mirman, M., Püschel, M., and Vechev, M. Fast and effective robustness certification. In Advances in Neural Information Processing Systems, pp. 10825–10836, 2018.
 Singh et al. (2019a) Singh, G., Gehr, T., Püschel, M., and Vechev, M. Robustness certification with refinement. ICLR, 2019a.
 Singh et al. (2019b) Singh, G., Gehr, T., Püschel, M., and Vechev, M. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3(POPL):41, 2019b.
 Szegedy et al. (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
 Tjeng et al. (2019a) Tjeng, V., Xiao, K., and Tedrake, R. Evaluating robustness of neural networks with mixed integer programming. ICLR, 2019a.
 Tjeng et al. (2019b) Tjeng, V., Xiao, K. Y., and Tedrake, R. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019b. URL https://openreview.net/forum?id=HyGIdiRqtm.
 Wang et al. (2018a) Wang, S., Chen, Y., Abdou, A., and Jana, S. Mixtrain: Scalable training of formally robust neural networks. arXiv preprint arXiv:1811.02625, 2018a.
 Wang et al. (2018b) Wang, S., Pei, K., Whitehouse, J., Yang, J., and Jana, S. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems, pp. 6369–6379, 2018b.
 Weng et al. (2018) Weng, T.W., Zhang, H., Chen, H., Song, Z., Hsieh, C.J., Boning, D., Dhillon, I. S., and Daniel, L. Towards fast computation of certified robustness for ReLU networks. In International Conference on Machine Learning, 2018.
 Wong & Kolter (2018) Wong, E. and Kolter, Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), pp. 5283–5292, 2018.
 Wong et al. (2018) Wong, E., Schmidt, F., Metzen, J. H., and Kolter, J. Z. Scaling provable adversarial defenses. Advances in Neural Information Processing Systems (NIPS), 2018.
 Xiao et al. (2019) Xiao, K. Y., Tjeng, V., Shafiullah, N. M. M., and Madry, A. Training for faster adversarial robustness verification via inducing reLU stability. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BJfIVjAcKm.
 Zhang et al. (2018) Zhang, H., Weng, T.W., Chen, P.Y., Hsieh, C.J., and Daniel, L. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems (NIPS), dec 2018.
 Zhang et al. (2019) Zhang, H., Zhang, P., and Hsieh, C.J. Recurjac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. AAAI Conference on Artificial Intelligence, 2019.
Appendix A Additional Theoretical Results and Proofs
a.1 The optimal convex relaxation
In this section, we study the optimal convex relaxation of the nonlinear constraint . If there is only one neuron, i.e., only a scalar output, the following proposition gives its optimal convex relaxation.
Proposition A.1.
Suppose the activation function is bounded from above and below. Let and be the greatest closed convex functions majored by and , respectively, i.e.,
(15)  
(16) 
Then we have,

Both and are continuous in .

where denotes the closed convex hull.