A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks

A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks

Hadi Salman
Microsoft Research AI
hadi.salman@microsoft.com &Greg Yang
Microsoft Research AI
gregyang@microsoft.com &Huan Zhang
huan@huan-zhang.com &Cho-Jui Hsieh
chohsieh@cs.ucla.edu &Pengchuan Zhang
Microsoft Research AI
Work done as part of the Microsoft AI Residency Program.

Verification of neural networks enables us to gauge their robustness against adversarial attacks. Verification algorithms fall into two categories: exact verifiers that run in exponential time and relaxed verifiers that are efficient but incomplete. In this paper, we unify all existing LP-relaxed verifiers, to the best of our knowledge, under a general convex relaxation framework. This framework works for neural networks with diverse architectures and nonlinearities and covers both primal and dual views of robust verification. We further prove strong duality between the primal and dual problems under very mild conditions. Next, we perform large-scale experiments, amounting to more than 22 CPU-years, to obtain exact solution to the convex-relaxed problem that is optimal within our framework for ReLU networks. We find the exact solution does not significantly improve upon the gap between PGD and existing relaxed verifiers for various networks trained normally or robustly on MNIST and CIFAR datasets. Our results suggest there is an inherent barrier to tight verification for the large class of methods captured by our framework. We discuss possible causes of this barrier and potential future directions for bypassing it.


A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks

  Hadi Salmanthanks: Work done as part of the Microsoft AI Residency Program. Microsoft Research AI hadi.salman@microsoft.com Greg Yang Microsoft Research AI gregyang@microsoft.com Huan Zhang UCLA huan@huan-zhang.com Cho-Jui Hsieh UCLA chohsieh@cs.ucla.edu Pengchuan Zhang Microsoft Research AI penzhan@microsoft.com


noticebox[b]Preprint. Work in progress.\end@float

1 Introduction

Neural networks (NNs) have had a lot of successes in tasks such as image classification (Krizhevsky et al., 2012; He et al., 2016) and speech recognition (Hinton et al., 2012). However, it has been shown that NNs are vulnerable to adversarial examples, for instance perturbed images which are visually indistinguishable from the original to a human, but which can change the NN’s classification dramatically (Szegedy et al., 2013; Goodfellow et al., 2015). This becomes a serious problem when NNs are applied to safety-critical applications, or those where security is a requirement. Such concern has prompted very active research in adversarial attack (designing adversarial examples) (Goodfellow et al., 2015; Carlini & Wagner, 2017; Chen et al., 2017), defense (train- or inference-time procedures to nullify effectiveness of adversarial examples) (Madry et al., 2017; Wong & Kolter, 2018; Raghunathan et al., 2018a), and verification (certifying that no small perturbations of a given input can cause the NN to change its prediction) (Katz et al., 2017; Dvijotham et al., 2018a; Weng et al., 2018). This paper is concerned with the last, robust verification.

Many recent works have proposed robust verification methods (Katz et al., 2017; Ehlers, 2017; Hein & Andriushchenko, 2017). Typically, such algorithms try to exactly find or upper bound an adversarial loss, defined to be the maximum gap between the network’s correct logit and any of its incorrect logits, over all possible norm-bounded perturbations of the input (see eq. 1). Alternatively, they exactly find or lower bound the minimum adversarial distortion , defined to be the radius of the largest ball (in a specified norm), around the input, that contains no adversarial examples. The algorithms devised so far can be roughly categorized as exact (Ehlers, 2017; Katz et al., 2017; Tjeng et al., 2019b) or relaxed methods (Wong & Kolter, 2018; Zhang et al., 2018; Singh et al., 2018), and among the latter, the most dominant category is convex relaxation, which tries to relax the exact adversarial optimization problem into a convex problem. Our first main contribution is to give a unifying treatment of convex relaxation used in a large number of recent algorithms (Fig. 1) and reveal the relationships between them. We further prove that these relaxed problems satisfy strong duality under very mild conditions, connecting relaxed verifiers in the primal space (Zhang et al., 2018; Weng et al., 2018) to those in the dual space (Dvijotham et al., 2018c; Wong & Kolter, 2018).

For any NN, our framework prescribes a unique tightest relaxation, whose optimal value presents a barrier for any method described by the framework. How close is to the true adversarial loss? In our second main contribution, we perform extensive experiments with deep ReLU networks to compute and compare with the LP-relaxed dual formulation of Wong & Kolter (2018) (which, as our framework will show, approximates ), the PGD attack of Madry et al. (2017), and the mixed integer linear programming (MILP) exact verifier of Tjeng et al. (2019b). Over different models, sizes, training methods, and datasets (MNIST and CIFAR-10), we find that our tightest relaxation only very slightly improves the lower bound of the minimum adversarial distortion found by Wong & Kolter (2018), especially when compared with the upper bound provided by the PGD attack, which is consistently 1.5 to 5 times larger. For networks trained using the LP-relaxed dual formulation of Wong & Kolter (2018), when is large and the robust error is also large, our tighest relaxation improves the certified robust error significantly in relative terms, but at most 7% in absolute terms. Furthermore, on both normally and PGD-trained MNIST classifiers, our tightest relaxation does not significantly close the gap in the robust error between the PGD lower bound or MILP exact answer on one hand, and the upper bound of Wong & Kolter (2018) on the other. Thus the convex relaxation barrier we discover theoretically indeed presents a significant challenge to existing algorithms empirically.

Figure 1: Relationship between existing relaxed algorithms and our framework. See main text.

2 Preliminaries and Related Work

Robustness to adversarial attacks.

Consider a multi-class classification network , where is the logit for the -th class. Given a nominal input , its nominal prediction , and a neighborhood , the following optimization problem determines if the network is robust to adversarial attacks within this neighborhood:


If the minimal value is positive for all , the network is robust at this point within . We refer the reader to Dvijotham et al. (2018c) and Qin et al. (2019) for other robust verification problems which can be solved with similar methods.

Exact verifiers and NP-completeness.

For ReLU networks (piece-wise linear networks in general), exact verifiers solve the robust verification problem (1) by typically employing MILP solvers (Cheng et al., 2017; Lomuscio & Maganti, 2017; Dutta et al., 2018; Fischetti & Jo, 2017; Tjeng et al., 2019a; Xiao et al., 2019) or Satisfiability Modulo Theories (SMT) solvers (Scheibler et al., 2015; Katz et al., 2017; Carlini et al., 2017; Ehlers, 2017). However, due to the NP-completeness for solving such a problem (Katz et al., 2017; Weng et al., 2018), it can be really challenging to scale these to large networks. It can take Reluplex (Katz et al., 2017) several hours to find the minimum distortion of an example for a ReLU network with 5 inputs, 5 outputs, and 300 neurons. Recent work by Tjeng et al. (2019b) uses MILP to exactly verify medium-size networks, but the verification time is very sensitive to how a network is trained; for example, it is fast for networks trained using the LP-relaxed dual formulation of Wong & Kolter (2018), but much slower for normally trained networks. A concurrent work by Xiao et al. (2019) trains networks with the objective of speeding up the MILP verification problem, but this compromises on the performance of the network.

Relaxed and efficient verifiers.

These verifiers solve a relaxed, but more computationally efficient, version of (1), and have been proposed from different perspectives. From the primal view, one can relax the nonlinearity in (1) into linear inequality constraints. This perspective has been previously explored as in the framework of “abstract transformers”  (Singh et al., 2018, 2019b, 2019a; Gehr et al., 2018; Mirman et al., 2018), via linear outer bounds of activation functions (Zhang et al., 2018; Weng et al., 2018; Wang et al., 2018b, a), or via interval bound propagation (Gowal et al., 2018). From the dual view, one can study the dual of the relaxed problem (Wong & Kolter, 2018; Wong et al., 2018) or study the dual of the original nonconvex verification problem (Dvijotham et al., 2018c, b; Qin et al., 2019). In this paper, we unify both views in a common convex relaxation framework for NN verification, clarifying their relationships (as summarized in Fig. 1).

Raghunathan et al. (2018b) formulates the verification of ReLU networks as a quadratic programming problem and then relaxes and solves this problem with a semidefinite programming (SDP) solver. While our framework does not cover this SDP relaxation, it is not clear to us how to extend the SDP relaxed verifier to general nonlinearities, for example max pooling, something which can be done in our framework on the other hand. Other verifiers have been proposed to certify via an intermediary step of bounding the local Lipschitz constant (Hein & Andriushchenko, 2017; Weng et al., 2018; Raghunathan et al., 2018a; Zhang et al., 2019). These are outside the scope of our framework.

Combining exact and relaxed verifiers, hybrid methods have shown some effectiveness (Bunel et al., 2018; Singh et al., 2019a). In fact, many exact verifiers also use relaxation as a subroutine to speed things up, and hence can be viewed as hybrid methods as well. In this paper, we are not concerned with such techniques but only focus on relaxed verifiers.

3 Convex Relaxation from the Primal View

Problem setting.

In this paper, we assume that the neighborhood is a convex set. An example of this is , which is the constraint on in the adversarial attack model. We also assume that is an -layer feedforward NN. For notational simplicity, we denote by and by . We define as,


where , , is the input, and are the weight matrix and bias vector of the linear layer, and is a (nonlinear) activation function like (leaky-)ReLU, the sigmoid family (including sigmoid, arctan, hyperbolic tangent, etc), and the pooling family (MaxPool, AvgPool, etc). Our results can be easily extended to networks with convolutional layers and skip connections as well, similar to what is done in Wong et al. (2018), as these can be seen as special forms of (2).

Consider the following optimization problem :


where the optimization domain is


If , , , and , then () is equivalent to problem (1). However, when we have better information about valid bounds and of , we can significantly narrow down the optimization domain and, as will be detailed shortly, achieve tighter solutions when we relax the nonlinearities. We denote the minimal value of by , or just when no confusion arises.

Obtaining lower and upper bounds by solving sub-problems.

This can be done by recursively solving () with specific choices of and , which is a common technique used in many works (Wong & Kolter, 2018; Dvijotham et al., 2018c). For example, one can obtain , a lower bound of , by solving ; this shows that one can estimate and inductively in . However, we may have millions of sub-problems to solve because practical networks can have millions of neurons. Therefore, it is crucial to have efficient algorithms to solve problem ().

General nonlinear specifications

are also of interest beyond the linear specification considered in (). These can be formulated as a special case of () by adding a nonlinear layer on top of the original network, and setting and . Because our framework can deal with general nonlinear layers, all the theoretical results in this paper can be applied to verification problems with nonlinear specifications as well.

Convex relaxation in the primal space.

Due to the nonlinear activation functions , the feasible set of () is nonconvex, which leads to the NP-completeness of the robust verification problem (Katz et al., 2017; Weng et al., 2018). One natural idea is to do convex relaxation of its feasible set. Specifically, one can relax the nonconvex equality constraint to convex inequality constraints, i.e.,


where ( ) is convex (concave) and satisfies for . We denote the feasible set of () by and its minimum by . Naturally, we have that is convex and .

For example, Ehlers (2017) proposed the following relaxation for the ReLU function :


and the following for MaxPool :


The optimal convex relaxation.

As a special case, we consider the optimal convex relaxation, where is the greatest convex function majored by , and is the smallest concave function majoring . They can be equivalently defined as


In Fig. 2, we show the optimal convex relaxation for several common activation functions. It is easy to see that (4) is the optimal convex relaxation for ReLU, but (5) is not optimal for the MaxPool function.

Figure 2: Optimal convex relaxations for common nonlinearities. For , the relaxation contains two linear segments and parts of the function. For ReLU and the step function, the optimal relaxations are written as 3 and 4 linear constraints, respectively. For , the light orange shadow indicates the pre-activation bounds for and , and the optimal convex relaxation is lower bounded by the function itself.

When is a nonlinear layer that has a vector output , the optimal convex relaxation may not have a simple analytic form as . Fortunately, if there is no interaction (as defined below) among the output neurons, the optimal convex relaxation can be given as a simple analytic form.

Definition 3.1 (non-interactive layer).

Let and be a nonlinear layer with input and output . For each output , let be the minimal set of ’s entries that affect , where . We call the layer non-interactive if the sets () are mutually disjoint.

Commonly used nonlinear activation layers are all non-interactive. It is obvious that all entry-wise nonlinear layers, such as (leaky-)ReLU and sigmoid, are non-interactive. A MaxPool layer with non-overlapping regions (stride no smaller than kernel size) is also non-interactive. Finally, any layer with scalar-valued output is non-interactive. When we treat a general nonlinear specification (as proposed in Qin et al. (2019)) as an additional nonlinear layer , this layer is automatically non-interactive. This nice property ensures that our framework can deal with very general specifications.

The optimal convex relaxation of a non-interactive layer has a simple analytic form as below.

Proposition 3.2.

If the layer is non-interactive, we have

where denotes the closed convex hull, and scalar-valued functions and are defined in (6) and (7), respectively.

We defer the proof of Proposition 3.2 to Appendix A.1.

Convex relaxations not included in Problem ().

We emphasize that by optimal, we mean the optimal convex relaxation of the single nonlinear constraint (see Proposition (3.2)) instead of the optimal convex relaxation of the nonconvex feasible set of the original problem (). In fact, for neural networks with more than two hidden layers (), the optimal convex relaxation of the nonconvex feasible set of problem () is a strict subset of the feasible set of problem (), even with the tightest bounds and the optimal choice of and in (6) and (7). It is possible to obtain other (maybe tighter) convex relaxations, but it comes with more assumptions on the nonlinear layers and more complex convex constraints.

For example, Raghunathan et al. (2018b) rewrites the ReLU nonlinearity as a quadratic constraint, and then proposes a semidefinite programming (SDP) relaxation for the resulting quadratic optimization problem. Problem () does not cover this SDP-relaxation. Sometimes Problem () provides tighter relaxation than the SDP-relaxation, e.g., the case when there is only one neuron in a layer, while sometimes the SDP-relaxation provides tighter relaxation than Problem (), e.g., the examples provided in Raghunathan et al. (2018b). The SDP-relaxation currently only works for ReLU nonlinearity. It is not clear to us how to extend the SDP-relaxed verifier to general nonlinearities. On the other hand, Problem () can handle any non-interactive nonlinear layer and any nonlinear specification.

Greedily solving the primal with linear bounds.

Before resorting to duality for large problems, it is possible to solve () by over-relaxing the problem to efficiently give a lower bound for the objective. This can be done by defining exactly one linear upper bound and one linear lower bound for each activation function in () as follows,

Given these bounds for each activation, the primal variables can be greedily bounded in a layer-by-layer manner: we can derive one linear upper and one linear lower bound of with respect to using the fact that , and that is linearly upper and lower bounded by and . Because a linear combination of linear bounds (coefficients are related to the entries in ) is a single linear bound, we can apply this technique again and replace with its upper and lower bounds with respect to , obtaining the bound for with respect to . Applying this repeatedly eventually leads to linear lower and upper bounds of with respect to the input .

This perspective covers AI (Gehr et al., 2018), Fast-Lin (Weng et al., 2018), DeepZ (Singh et al., 2018) and Neurify (Wang et al., 2018b), where the proposed linear lower bound has the same slope as the upper bound, i.e., . The resulting shape is referred to as a zonotope in Gehr et al. (2018) and Singh et al. (2018). In CROWN (Zhang et al., 2018) and DeepPoly (Singh et al., 2019b), this restriction is lifted and they can achieve better verification results than Fast-Lin and DeepZ. Fig. 1 summarizes the relationships between these algorithms. Importantly, each of these works has its own merits on solving the verification problem; our focus here is to give a unified view on how they over-relax  in our framework. See Appendix A.2 for more discussions and other related algorithms.

4 Convex Relaxation from the Dual View

We now tackle the verification problem from the dual view and show its connection to the primal view.

Strong duality for the convex relaxed problem.

As in Wong & Kolter (2018), we introduce the dual variables for () and write its Lagrangian dual as


Note that for any choice of , and , the value of the dual function lower bounds the optimal value of (), an thus we have


This property is known as weak duality (Boyd & Vandenberghe, 2004). In the following, we prove that strong duality between () and (9) holds under some mild conditions.

Theorem 4.1.

If, for each , both and have a finite Lipschitz constant in the domain , then strong duality holds between () and (9), i.e.,

The assumption of the finite Lipschitz constant is so weak that one can always construct such and as long as is bounded in the domain . Notice that we do not make additional geometric assumptions (like Slater’s condition) in Theorem (4.1). Instead, we make use of geometric properties of NNs, i.e., their linear stability against perturbations in the layer-wise computation, to obtain our strong duality results. We present the linear stability analysis in Lemma A.1 and provide the strong duality proof in Appendix A.3.

The optimal dual relaxation.

If we restrict ourselves to the dual of the convex relaxed problem (), the optimal (tightest) bound the dual problem can provide is exactly the bound the optimal convex relaxation ((6) and (7)) provides, thanks to the strong duality.

One approach to possibly obtain a tighter dual problem is to directly study the Lagrangian dual of the original problem (),


which was first proposed in Dvijotham et al. (2018c). Note that for any choice of and , the value of the dual function lower bounds the optimal value of (). Thus,


It would seem to be strictly better than (9). Unfortunately, the following theorem shows that the dual problem (11) can do no better than the dual of the convex relaxation (9).

Theorem 4.2.

Assume that the nonlinear layer is non-interactive and and are defined in (6) and (7), respectively. Then the lower bound provided by the dual of the convex-relaxed problem (9) and provided by the dual of the original problem (11) are the same, i.e.,

Proof sketch.

First, we prove that for any , and , with , we have


where and . Therefore, the dual problem (9) can be rewritten as an unconstrained optimization problem as


Second, we prove that for any and , we have


Combining (11), (13) and (14), we obtain . ∎

The complete proof is in Appendix A.4. From (13) and (14), we can see that these two dual problems not only share the optimal value, but also share the same objective function! We would like to point out that Theorem 2 in Dvijotham et al. (2018c) is a special case of our Theorem 4.2, when applied to ReLU networks. Our proof makes use of the Fenchel-Moreau theorem (Theorem 12.2 in Rockafellar (2015)) to deal with general nonlinearities, which is different from that in Dvijotham et al. (2018c).

Theorem 4.2 combined with the strong duality result of Theorem 4.1 implies that the primal relaxation () and the two kinds of dual relaxations, (9) and (11), are all blocked by the same barrier — the bound that the optimal convex relaxation ((6) and (7)) provides.

Corollary 4.3.

Suppose that the activation functions () are non-interactive and are among the following functions: piece-wise linear functions (e.g., ReLU and stair-case functions111Note that we only require and defined in (6) and (7) be Lipschitz continuous, while itself may not be Lipschitz.), ELU, sigmoid, arctan, hyperbolic tangent, polynomials and common pooling functions (e.g., MaxPool and AvgPool). Assume that and are defined in (6) and (7), respectively. Then we have that the lower bound provided by the primal relaxation () and provided by the dual relaxation (11) are the same, i.e.,

We point out that the constraint on the activation function is to ensure the strong duality between () and (9) when and are defined in (6) and (7), respectively. This constraint is so weak that it is satisfied by nearly all commonly used activation functions, far more than the functions listed in Corollary 4.3.

Which problem to solve in practice?

Thanks to the strong duality, the same lower bound can be achieved from both the primal and the dual problems, and thus we have the freedom to choose the problem to solve. When the relaxed upper and lower bounds, i.e., and , are piece-wise linear (e.g. (4) for ReLU networks), both the primal and dual problems are linear programs and can be efficiently solved by existing LP solvers (which is what we use in the coming sections). In other cases, we recommend to solve the dual problem (11) for two reasons. First, the primal relaxed problem () is a constrained optimization problem, and its constraints may not have a simple analytic form when and are not piecewise linear; see examples in Fig. 2. On the contrary, the dual problem (11) can be framed as an unconstrained optimization problem and its objective function has a simple analytic form for some common activation functions (Dvijotham et al., 2018c). Second, the optimization process of (11) can be stopped anytime to give a lower bound of , thanks to weak duality, but this is not true of the primal view. Of course, and must be in the form of (6) and (7) to achieve the optimal value.

Greedily solving the dual with linear bounds.

When the relaxed bounds and are linear, i.e.,

the dual objective (9) can be approximately solved (lower bounded) by a greedy algorithm. For }, this greedy algorithm fixes the already obtained sub-optimal solution of , then computes a sub-optimal solution for by maximizing the part of the objective function that only involves . We provide the algorithm details in Appendix A.5.

It turns out that this greedy algorithm exactly recovers the algorithm proposed in Wong & Kolter (2018) when

where represents the slope of the lower bound. When , the greedy algorithm also recovers Fast-Lin (Weng et al., 2018), which explains the arrow from Wong & Kolter (2018) to Weng et al. (2018) in Fig. 1. When is chosen adaptively as in CROWN (Zhang et al., 2018), the greedy algorithm then recovers CROWN, which explains the arrow from Wong & Kolter (2018) to Zhang et al. (2018) in Fig. 1. See Appendix A.2 for more discussions on the relationship between the primal and dual greedy solvers.

5 Optimal LP-relaxed Verification

In the previous sections, we presented a framework that subsumes all existing convex-relaxed verification algorithms except that of Raghunathan et al. (2018b). For ReLU networks, being piece-wise linear, these correspond exactly to the set of all existing LP-relaxed algorithms, as discussed above. We showed the existence of a barrier, , that limits all such algorithms. Is this just theoretical babbling or is this barrier actually problematic in practice?

In the next section, we perform extensive experiments on deep ReLU networks, evaluating the tightest convex relaxation afforded by our framework (denoted LP-all) against the algorithm of Wong & Kolter (2018) (denoted LP-greedy) as well as another algorithm LP-last, intermediate in speed and accuracy between them. Both LP-greedy and LP-last solve the bounds by setting the dual variables heuristically (see previous section), but LP-greedy solves the adversarial loss in the same manner while LP-last solves this final LP exactly. We also compare them with the opposite bounds provided by PGD attack (Madry et al., 2017), as well as MILP exact results (Tjeng et al., 2019b).

For the rest of the main text, we are only concerned with ReLU networks, so () subject to (4) is in fact an LP.

5.1 LP-all Implementation Details

In order to exactly solve the tightest LP-relaxed verification problem of a ReLU network, two steps are required: (A) obtaining the tightest pre-activation upper and lower bounds of all the neurons in the NN, excluding those in the last layer, then (B) solving the LP-relaxed verification problem exactly for the last layer of the NN.

Step A: Obtaining Pre-activation Bounds.

This can be done by solving sub-problems of the orginial relaxed problem () subject to (4). Given a NN with layers, for each layer , we obtain a lower (resp. upper) bound (resp. ) of , for all neurons . We do this by setting

in () and computing the exact optimum.

However, we need to solve an LP for each neuron, and practical networks can have millions of them. We utilize the fact that in each layer , computing the bounds and for each can proceed independently in parallel. Indeed, we design a scheduler to do so on a cluster with 1000 CPU-nodes. See Appendix C for details.

Step B: Solving the LP-relaxed Problem for the Last Layer.

After obtaining the pre-activation bounds on all neurons in the network using step (A), we solve the LP in () subject to (4) for all obtained by setting

again in () and computing the exact optimum.

Here, is the true label of the data point at which we are verifying the network. We can certify the network is robust around iff the solutions of all such LPs are positive, i.e. we cannot make the true class logit lower than any other logits. Again, note that these LPs are also independent of each other, so we can solve them in parallel.

Given any , LP-all follows steps (A) then (B) to produce a certificate whether the network is robust around a given datapoint or not. LP-last on the other hand solves only step (B), and instead of doing step (A), it finds the preactivation bounds heuristically as in Wong & Kolter (2018).

6 Experiments

We conduct two experiments to assess the tightness of LP-all: 1) finding certified bounds on the robust error of several NN classifiers, 2) finding certified bounds on the minimum adversarial distortion using different algorithms. All experiments are conducted on MNIST and/or CIFAR-10 datasets.


We conduct experiments on a range of ReLU-activated feedforward networks. MLP-A and MLP-B refer to multilayer perceptrons: MLP-A has 1 hidden layer with 500 neurons, and MLP-B has 2 hidden layers with 100 neurons each. CNN-small, CNN-wide-k, and CNN-deep-k are the ConvNet architectures used in Wong et al. (2018). Full details are in Appendix B.1.

Training Modes.

We conduct experiments on networks trained with a regular cross-entropy (CE) loss function and networks trained to be robust. These networks are identified by a prefix corresponding to the method used to train them: LPd when the LP-relaxed dual formulation of Wong & Kolter (2018) is used for robust training, Adv when adversarial examples generated using PGD are used for robust training, as in Madry et al. (2017), and Nor when the network is normally trained using the CE loss function. Training details are in Appendix B.2.

Experimental Setup.

We run experiments on a cluster with 1000 CPU-nodes. The total run time amounts to more than 22 CPU-years. Appendix C provides additional details about the computational resources and the scheduling scheme used, and Appendix D provides statistics of the verification time in these experiments.

max width=0.9 Network Test Error Lower Bound Upper Bound PGD MILP MILP LP-all LP-greedy Adv-MLP-B 0.03 1.53% 3.67% 4.16% 5.78% 10.04% 13.40% Adv-MLP-B 0.05 1.62% 5.79% 5.99% 11.38% 23.29% 33.09% Adv-MLP-B 0.1 3.33% 15.44% 16.12% 34.37% 61.59% 71.34% Adv-MLP-A 0.1 4.18% 11.51% 14.36% 30.81% 60.14% 67.50% Nor-MLP-B 0.02 2.05% 9.96% 10.14% 13.48% 26.41% 35.11% Nor-MLP-B 0.03 2.05% 19.88% 20.02% 48.67% 65.70% 75.85% Nor-MLP-B 0.05 2.05% 51.31% 51.31% 94.04% 97.95% 99.39% LPd-MLP-B 0.1 4.09% 13.23% 14.45% 14.45% 17.24% 18.32% LPd-MLP-B 0.2 15.72% 33.38% 36.33% 36.33% 37.50% 41.67% LPd-MLP-B 0.3 39.22% 56.82% 59.85% 59.85% 60.17% 66.85% LPd-MLP-B 0.4 67.97% 81.66% 83.17% 83.17% 83.62% 87.89%

Table 1: Certified bounds on the robust error on the test set of MNIST for normally and robustly trained networks. The prefix of each network corresponds to the training method used: Adv for PGD training (Madry et al., 2017), Nor for normal CE loss training, and LPd when the LP-relaxed dual formulation of Wong & Kolter (2018) is used for robust training.

6.1 Certified Bounds on the Robust Error

Table 1 presents the clean test errors and (upper and lower) bounds on the true robust errors for a range of classifiers trained with different procedures on MNIST. For both Adv- and LPd-trained networks, the in Table 1 denotes the -norm bound used for training and robust testing; for Normally-trained networks, is only used for the latter.

Lower bounds on the robust error are calculated by finding adversarial examples for inputs that are not robust. This is done by using PGD, a strong first-order attack, or using MILP (Tjeng et al., 2019b). Upper bounds on the robust error are calculated by providing certificates of robustness for input that is robust. This is done using MILP, the dual formulation (LP-greedy) presented by Wong & Kolter (2018), or our LP-all algorithm.

For the MILP results, we use the code accompanying the paper by Tjeng et al. (2019b). We run the code in parallel on a cluster with 1000 CPU-nodes, and set the MILP solver’s time limit to 3600 seconds. Note that this time limit is reached for Adv and Nor, and therefore the upper and lower bounds are separated by a gap that is especially large for some of the Normally trained networks. On the other hand, for LPd-trained networks, the MILP solver finishes within the time limit, and thus the upper and lower bounds match.

For all Normally and Adv-trained networks, we see that the certified upper bounds using LP-greedy and LP-all are very loose when we compare the gap between them to the lower bounds found by PGD and MILP. As a sanity check, note that LP-all gives a tighter bound than LP-greedy in each case, as one would expect. Yet this improvement is not significant enough to close the gap with the lower bounds.

This sanity check also passes for LPd-trained networks, where the LP-greedy-certified robust error upper bound is, as expected, much closer to the true error (given by MILP here) than for other networks. For , the improvement of LP-all-certified upper bound over LP-greedy is at most modest, and the PGD lower bound is tighter to the true error. For large , the improvement is much more significant in relative terms, but the absolute improvement is only . In this large regime, however, both the clean and robust errors are quite large, so the tightness of LP-all is less useful.

Figure 3: The median percentage gap between the convex-relaxed algorithms (LP-all, LP-last, and LP-greedy) and PGD estimates of the minimum adversarial distortion on ten samples of MNIST. The error bars correspond to 95% confidence intervals. We highlight the and gaps between the value estimated by PGD, and those estimated by the LP-relaxed algorithms. For more details, please refer to Table 2 in Appendix E.2.
Figure 4: The median percentage gap of minimum adversarial distortion for CIFAR-10, in the same format as Fig. 3. For more details, please refer to Table 3 in Appendix E.2.

6.2 Certified Bounds on the Minimum Adversarial Distortion

We are interested in searching for the minimum adversarial distortion , which is the radius of the largest ball in which no adversarial examples can be crafted. An upper bound on is calculated using PGD, and lower bounds are calculated using LP-greedy, LP-last, or our LP-all, all via binary search. Since solving LP-all is expensive, we find the -bounds only for ten samples of the MNIST and CIFAR-10 datasets. See Appendix E.1 for details.

Fig. 3 and 4 show the median percentage gap (defined in Appendix E.2) between the convex-relaxed algorithms and PGD estimates of for MNIST and CIFAR, respectively. Details are reported in Tables 2 and 3 in Appendix E.2.

On MNIST, the results show that for all networks trained Normally or via Adv, the certified lower bounds on are 1.5 to 5 times smaller than the upper bound found by PGD; for LPd trained networks, below 1.5 times smaller. On CIFAR-10, the bounds are between 1.5 and 2 times smaller across all models.

The smaller gap for LPd is of course as expected following similar observations in prior work (Wong & Kolter, 2018; Tjeng et al., 2019b). Furthermore, the improvement of LP-all and LP-last over LP-greedy is not significant enough to close the gap with the PGD upper bound. Note that similar results hold as well for randomly initialized networks (no training). To avoid clutter, we report these in Appendix F.

7 Discussion

In this work we first presented a convex relaxation framework that unifies all previous LP-relaxed verifiers. Then we performed extensive experiments to show that even the optimal convex relaxation for ReLU networks in this framework cannot obtain tight bounds on robust error in all cases we consider here. Thus any method will face a convex relaxation barrier as soon as it can be described by our framework.

Note that different applications have different requirements for the tightness of the verification, so our barrier could be a problem for some but not for others. In so far as the ultimate goal of robust verification is to construct a training method to lower certified error, this barrier is not necessarily problematic — some such method could still produce networks for which convex relaxation as described by our framework produces accurate robust error bounds. An example is the recent work of Gowal et al. (2018) which shows that interval bound propagation, which often leads to loose certification bounds, can still be used for verified training, and is able to achieve state-of-the-art verified accuracy when carefully tuned. However, without a doubt, in all cases, tighter estimates should lead to better results, and we reveal a definitive ceiling on most current methods.

How to bypass the barrier?

The primal problem in our framework () has several possible sources of loosenes:

  1. We relax the nonlinearity on a box domain . This relaxation is simple to perform, but might come at the cost of losing the correlation between the coordinates of and of obtaining a looser relaxation.

  2. We solve for the bounds recursively, and we incur some gap for every recursion.

  3. In the specific case of ReLU, we lose a bit every time we relax over an unstable neuron; one possible future direction is to combine branch-and-bound with convex relaxation to strategically split the domains of unstable neurons.

Any method that improves on any of the above issues can possibly bypass the barrier. On the other hand, exact verifiers (Katz et al., 2017; Ehlers, 2017), SDP-based verifiers (Raghunathan et al., 2018b, a), Lipschitz-constant-based verifiers (Zhang et al., 2019), and hybrid approaches (Bunel et al., 2018; Singh et al., 2019a) do not fall under the purview of our framework. In general, none of them are strictly better than the convex relaxation approach, sacrificing either speed or accuracy. However, it would be fruitful to consider combinations of these methods in the future, as done in Singh et al. (2019a).

We hope our work will foster much thought in the community toward new relaxation paradigms for tight robust verification.


  • Boyd & Vandenberghe (2004) Boyd, S. and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
  • Bunel et al. (2018) Bunel, R. R., Turkaslan, I., Torr, P., Kohli, P., and Mudigonda, P. K. A unified view of piecewise linear neural network verification. In Advances in Neural Information Processing Systems, pp. 4795–4804, 2018.
  • Carlini & Wagner (2017) Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57. IEEE, 2017.
  • Carlini et al. (2017) Carlini, N., Katz, G., Barrett, C., and Dill, D. L. Provably minimally-distorted adversarial examples. arXiv preprint arXiv:1709.10207, 2017.
  • Chen et al. (2017) Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., and Hsieh, C.-J. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15–26. ACM, 2017.
  • Cheng et al. (2017) Cheng, C.-H., Nührenberg, G., and Ruess, H. Maximum resilience of artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis, pp. 251–268. Springer, 2017.
  • Diamond & Boyd (2016) Diamond, S. and Boyd, S. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1–5, 2016.
  • Domahidi et al. (2013) Domahidi, A., Chu, E., and Boyd, S. ECOS: An SOCP solver for embedded systems. In European Control Conference (ECC), pp. 3071–3076, 2013.
  • Dutta et al. (2018) Dutta, S., Jha, S., Sankaranarayanan, S., and Tiwari, A. Output range analysis for deep feedforward neural networks. In NASA Formal Methods Symposium, pp. 121–138. Springer, 2018.
  • Dvijotham et al. (2018a) Dvijotham, K., Garnelo, M., Fawzi, A., and Kohli, P. Verification of deep probabilistic models. CoRR, abs/1812.02795, 2018a. URL http://arxiv.org/abs/1812.02795.
  • Dvijotham et al. (2018b) Dvijotham, K., Gowal, S., Stanforth, R., Arandjelovic, R., O’Donoghue, B., Uesato, J., and Kohli, P. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265, 2018b.
  • Dvijotham et al. (2018c) Dvijotham, K., Stanforth, R., Gowal, S., Mann, T., and Kohli, P. A dual approach to scalable verification of deep networks. UAI, 2018c.
  • Ehlers (2017) Ehlers, R. Formal verification of piece-wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis, pp. 269–286. Springer, 2017.
  • Fischetti & Jo (2017) Fischetti, M. and Jo, J. Deep neural networks as 0-1 mixed integer linear programs: A feasibility study. arXiv preprint arXiv:1712.06174, 2017.
  • Gehr et al. (2018) Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., and Vechev, M. AI 2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP), 2018.
  • Goodfellow et al. (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. ICLR, 2015.
  • Gowal et al. (2018) Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715, 2018.
  • He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  • Hein & Andriushchenko (2017) Hein, M. and Andriushchenko, M. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems (NIPS), pp. 2266–2276, 2017.
  • Hinton et al. (2012) Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.-r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82–97, 2012.
  • Katz et al. (2017) Katz, G., Barrett, C., Dill, D. L., Julian, K., and Kochenderfer, M. J. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pp. 97–117. Springer, 2017.
  • Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  • Lomuscio & Maganti (2017) Lomuscio, A. and Maganti, L. An approach to reachability analysis for feed-forward relu neural networks. arXiv preprint arXiv:1706.07351, 2017.
  • Madry et al. (2017) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  • Mirman et al. (2018) Mirman, M., Gehr, T., and Vechev, M. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, pp. 3575–3583, 2018.
  • Qin et al. (2019) Qin, C., Dvijotham, K. D., O’Donoghue, B., Bunel, R., Stanforth, R., Gowal, S., Uesato, J., Swirszcz, G., and Kohli, P. Verification of non-linear specifications for neural networks. ICLR, 2019.
  • Raghunathan et al. (2018a) Raghunathan, A., Steinhardt, J., and Liang, P. Certified defenses against adversarial examples. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1801.09344, 2018a.
  • Raghunathan et al. (2018b) Raghunathan, A., Steinhardt, J., and Liang, P. S. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp. 10900–10910, 2018b.
  • Rockafellar (2015) Rockafellar, R. T. Convex analysis. Princeton university press, 2015.
  • Scheibler et al. (2015) Scheibler, K., Winterer, L., Wimmer, R., and Becker, B. Towards verification of artificial neural networks. In MBMV, pp. 30–40, 2015.
  • Singh et al. (2018) Singh, G., Gehr, T., Mirman, M., Püschel, M., and Vechev, M. Fast and effective robustness certification. In Advances in Neural Information Processing Systems, pp. 10825–10836, 2018.
  • Singh et al. (2019a) Singh, G., Gehr, T., Püschel, M., and Vechev, M. Robustness certification with refinement. ICLR, 2019a.
  • Singh et al. (2019b) Singh, G., Gehr, T., Püschel, M., and Vechev, M. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3(POPL):41, 2019b.
  • Szegedy et al. (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  • Tjeng et al. (2019a) Tjeng, V., Xiao, K., and Tedrake, R. Evaluating robustness of neural networks with mixed integer programming. ICLR, 2019a.
  • Tjeng et al. (2019b) Tjeng, V., Xiao, K. Y., and Tedrake, R. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019b. URL https://openreview.net/forum?id=HyGIdiRqtm.
  • Wang et al. (2018a) Wang, S., Chen, Y., Abdou, A., and Jana, S. Mixtrain: Scalable training of formally robust neural networks. arXiv preprint arXiv:1811.02625, 2018a.
  • Wang et al. (2018b) Wang, S., Pei, K., Whitehouse, J., Yang, J., and Jana, S. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems, pp. 6369–6379, 2018b.
  • Weng et al. (2018) Weng, T.-W., Zhang, H., Chen, H., Song, Z., Hsieh, C.-J., Boning, D., Dhillon, I. S., and Daniel, L. Towards fast computation of certified robustness for ReLU networks. In International Conference on Machine Learning, 2018.
  • Wong & Kolter (2018) Wong, E. and Kolter, Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), pp. 5283–5292, 2018.
  • Wong et al. (2018) Wong, E., Schmidt, F., Metzen, J. H., and Kolter, J. Z. Scaling provable adversarial defenses. Advances in Neural Information Processing Systems (NIPS), 2018.
  • Xiao et al. (2019) Xiao, K. Y., Tjeng, V., Shafiullah, N. M. M., and Madry, A. Training for faster adversarial robustness verification via inducing reLU stability. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BJfIVjAcKm.
  • Zhang et al. (2018) Zhang, H., Weng, T.-W., Chen, P.-Y., Hsieh, C.-J., and Daniel, L. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems (NIPS), dec 2018.
  • Zhang et al. (2019) Zhang, H., Zhang, P., and Hsieh, C.-J. Recurjac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. AAAI Conference on Artificial Intelligence, 2019.

Appendix A Additional Theoretical Results and Proofs

a.1 The optimal convex relaxation

In this section, we study the optimal convex relaxation of the nonlinear constraint . If there is only one neuron, i.e., only a scalar output, the following proposition gives its optimal convex relaxation.

Proposition A.1.

Suppose the activation function is bounded from above and below. Let and be the greatest closed convex functions majored by and , respectively, i.e.,


Then we have,

  1. Both and are continuous in .

where denotes the closed convex hull.


1. By the boundedness of on , we know that the effective domain of and is . By definition (15) and (16), and are closed convex functions. By Theorem 10.2 in Rockafellar (2015), we know that both and are continuous in , so is .

2. We first decompose the left-hand-side into 3 terms:

Let and . For the first term, by definition (15) we have

For the second term, by definition (16) we have

For the third term, we have