A ModelBased DerivativeFree Approach to BlackBox Adversarial Examples: BOBYQA
Abstract
We demonstrate that modelbased derivative free optimisation algorithms can generate adversarial targeted misclassification of deep networks using fewer network queries than nonmodelbased methods. Specifically, we consider the blackbox setting, and show that the number of networks queries is less impacted by making the task more challenging either through reducing the allowed perturbation energy or training the network with defences against adversarial misclassification. We illustrate this by contrasting the BOBYQA algorithm Powell (2009) with the stateoftheart modelfree adversarial targeted misclassification approaches based on genetic Alzantot et al. (2019), combinatorial Moon et al. (2019), and directsearch Andriushchenko et al. (2019) algorithms. We observe that for high energy perturbations on networks, the aforementioned simpler modelfree methods require the fewest queries. In contrast, the proposed BOBYQA based method achieves stateoftheart results when the perturbation energy decreases, or if the network is trained against adversarial perturbations.
1 Introduction
Deep neural networks (NNs) achieve stateoftheart performance on a growing number of applications such as acoustic modelling, image classification, and fake news detection Hinton et al. (2012); He et al. (2015); Monti et al. (2019) to name but a few. Alongside their growing application, there is a literature on the robustness of deep nets which shows that it is often possible to generate images with subtle perturbations, referred to as adversarial examples Szegedy et al. (2014); Goodfellow et al. (2015), to the input of a network resulting in its performance being severely degraded; for example, see Dalvi et al. (2004); Kurakin et al. (2017); Sitawarin et al. (2018); Eykholt et al. (2018); Yuan et al. (2019) concerning the usecase of self driving cars.
Methods to generate these adversarial examples are classified according to two main criteria Yuan et al. (2019).
 Adversarial Specificity

establishes what the aim of the adversary is. In nontargeted attacks, the method perturbs the image in such a way that it is misclassified into any different category than the original one. While in targeted settings, the adversary specifies a category into which an image has to be misclassified.
 Adversary’s Knowledge

defines the amount of information available to the adversary. In Whitebox settings the adversary has complete knowledge of the network architecture and weights, while in the Blackbox setting the adversary is only able to obtain the preclassification outpupt vector for a limited number of inputs. The Whitebox setting allows for the use of gradients of a missclassification objective to efficiently compute the adversarial example Goodfellow et al. (2015); Carlini and Wagner (2017); Chen et al. (2018), while the same optimization formulation of the Blackbox setting requires use of a derivative free approach Narodytska and Kasiviswanathan (2017); Chen et al. (2017); Ilyas et al. (2018); Alzantot et al. (2019).
The generation of blackbox targeted adversarial examples for deep NNs has been extensively studied in a setting initially proposed by Chen et al. (2017) where:

the adversarial example is found by solving an optimisation problem designed to change the original classification of a specific input to a specific alternative.

the perturbation, which causes the network to change the classification, has entries bounded in magnitude by a specified infinity norm (maximum entry magnitude).

the number of queries to the NN needed to generate the adversarial example should be as small as possible.
The ZerothOrderOptimization (ZOO) Chen et al. (2017) introduced DFO methods for computing adversarial examples in the blackbox setting, specifically using a coordinate descent optimization algorithm. At the time this was a substantial departure from methods for the blackbox setting which train a proxy NN and then employ gradient based methods for whitebox attacks on the proxy network Papernot et al. (2017); Tu et al. (2019); such methods are especially effective when numerous adversarial examples will be computed, but require substantially more network queries than the methods designed for misclassifying individual examples. Following the introduction of ZOO, there have been numerous improvements using other modelfree DFO based approaches, see Alzantot et al. (2019); Moon et al. (2019); Andriushchenko et al. (2019). Specifically, GenAttack Alzantot et al. (2019) is a genetic algorithm, COMBI Moon et al. (2019) is a directsearch method that explores the vertices of the perturbation energy, and SQUARE Andriushchenko et al. (2019) is a randomized directsearch method.
In this manuscript we consider an alternative modelbased DFO method based on BOBYQA Powell (2009) which explicitly develops models that approximate the loss function in the optimisation problem and minimises the models using methods from continuous optimisation. By considering adversarial perturbations to three NNs trained on different datasets (MNIST, CIFAR10, and ImageNet), we show that for the modelfree methods Alzantot et al. (2019); Moon et al. (2019); Andriushchenko et al. (2019) the number of evaluation of the NN grows more rapidly as the maximum perturbation energy decreases than does the method built upon BOBYQA. As a consequence GenAttack, COMBI and SQUARE are preferable for large values of the maximum perturbation energy and BOBYQA for smaller values. As an example Figure 1 illustrates how the BOBYQA based algorithm compares to GenAttack, COMBI, and SQUARE when considering a net either normally or adversarially trained on CIFAR10 with different maximum perturbation energies.
We observe the intuitive principle that directsearch methods are effective to misclassify NNs with high perturbation energies, while in more challenging settings it is preferable to use more sophisticated modelbased methods, like ours. Modelbased approaches will further challenge defences to adversarial missclassification Dhillon et al. (2018); Wang et al. (2019), and in so doing will lead to improved defences and more robust networks. Modelbased DFO is a well developed area, and we expect further improvements are possible through a more extensive investigation of these approaches.
2 Adversarial Examples Formulated as an Optimisation Problem
Consider a classification operator from input space to output space of classes. A targeted adversarial perturbation to an input has the property that it changes the classification to a specified target class , i.e and . Herein we follow the formulation by Alzantot et al. (2019). Given: an image X, a maximum energy budget , and a suitable loss function , then the task of computing the adversarial perturbation can be cast as an optimisation problem such as
(1)  
where the final two inequality constraints are due to the input entries being restricted to . Denoting the preclassification output vector by , i.e. , then the misclassification of X to target label is achieved by if . In Carlini and Wagner (2017); Chen et al. (2017); Alzantot et al. (2019) they determined
(2) 
to be the most effective loss function for computing in (1), and we also employ this choice throughout our experiments.
3 Derivative Free Optimisation for Adversarial Examples
Derivative Free Optimisation is a well developed field with numerous types of algorithms, see Conn et al. (2009) and Larson et al. (2019) for reviews on DFO principles and algorithms. Examples of classes of such methods include: direct search methods such as simplex, modelbased methods, hybrid methods such as finite differences or implicit filtering, as well as randomized variants of the aforementioned and methods specific to convex or noisy objectives. The optimization formulation in Section 2 is amenable to virtually all DFO methods, making it unclear which of the algorithms to employ. Methods which have been trialled include: the finite difference based ZOO attack Chen et al. (2017), a combinatorial direct search of the perturbation energy constraint method COMBI Moon et al. (2019), a genetic direct search method GenAttack Alzantot et al. (2019), and most recently a randomized directsearch method Andriushchenko et al. (2019). Notably missing from the aforementioned list are modelbased methods.
Given a set of samples with , modelbased DFO methods start by identifying the minimiser of the objective among the samples at iteration , . Following this, a model for the objective function is constructed, typically centred around the minimizer. In its simplest form one uses a polynomial approximation to the objective, such as a quadratic model centred in
(3) 
with , c, , and being also symmetric. In a whitebox setting one would set and , but this is not feasible in the blackbox setting as we do not have access to the derivatives of the objective function. Thus c and M are usually defined by imposing interpolation conditions
(4) 
and when (i.e. the system of equations is underdetermined) other conditions are introduced according to which algorithm is considered. The objective model (3) is considered to be a good estimate of the objective in a neighbourhood referred to as a trust region. Once the model is generated, the update step p is computed by solving the trust region problem
(5) 
where is the radius of the region where we believe the model to be accurate, for more details see Nocedal and Wright (2006). The new point is added to and a prior point is potentially removed. Herein we consider an exemplary
Bobyqa
The BOBYQA algorithm, introduced in Powell (2009), updates the parameters of the model and M, in each iteration in such a way as to minimise the change in the quadratic term between iterates while otherwise fitting the sample values:
(6)  
(7) 
with and initialised as the zero matrix. When the number of parameters then the model is considered as linear with set as zero. We further allow only queries at each implementation of BOBYQA, since after the model is generated few iterations are needed to find the minimum.
3.1 Computational Scalability and Efficiency
For improved computational scalability and efficiency, we do not solve (1) for directly, but instead use domain subsampling and hierarchical liftings: domain subsampling iteratively sweeps over batches of variables, see (8), while hierarchical liftings clusters and perturbs variables simultaneously, see (12).
Domain SubSampling
The simplest version of domain subsampling consists of partitioning input dimension into smaller disjoint domains; for example, domains of size which are disjoint and which cover all of . Rather than solving (1) for directly, for each of one sequentially solves for which are only nonzero for entries in . The resulting subdomain perturbations are then summed to generate the full perturbation , see Figure 2 as an example. That is, the optimisation problem (1) is adapted to repeatedly looping over :
(8)  
where the may be reinitialised; in particular following each loop over which occurs at .
We considered three possible ways of selecting the domains

In Random Sampling we consider at each iteration a different random subsamplings of the domain, .

In Ordered Sampling we generate a random disjoint partitioning of the domain. Once each variable has been optimised over once a new partitioning is generated.

In Variance Sampling we choose to select in decreasing order of local variance of , the variance in intensity among the 8 neighbouring variables (e.g. pixels) in the same colour channel. We further reinitialise after each loop through .
In Figure 3 we compare how these different subsampling techniques perform when generating adversarial example for the MNIST and CIFAR10 dataset. It can be observed that variance sampling consistently performs better than random and ordered sampling. This suggest that pixels belonging to highcontrast regions are more influential than the ones in a lowcontrast one, and hence variance sampling is the preferable ordering.
Hierarchical Lifting
When the domain is very high dimensional, working on single pixels is not efficient as the above described method would imply modifying only a very small proportion of the image; for instance, we will choose even when is almost threehundredthousand. Thus to perturb wider portions of the image, we consider a hierarchy of liftings as in the ZOO attack presented in Chen et al. (2017). We seek an adversarial example by optimising over increasingly higher dimensional spaces at each step referred here as level lifted to the image space. As an illustration, Figure 4 shows that hierarchical lifting has a significant impact on the minimisation of the loss function.
At each level we consider a linear lifting and find a level perturbation which is added to the full perturbation , according to
(9) 
where is initialised as and the level perturbations of the previous layers are considered as fixed. Moreover, we impose that at each level, the grid has to double in refinement, i.e. . An example of how this works is illustrated in Figure 5.
When generating our adversarial examples, we considered two kind of liftings. The first kind of liftings is based on interpolation operations; a sorting matrix is applied such that every index of is uniquely associated to a node of a coarse grid masked over the original image. Afterwards, an interpolation is implemented over the values in the coarse grid, i.e. . The second kind of liftings, instead, forces the perturbation to be highfrequency since there is several literature on these perturbations being the most effective Guo et al. (2018); Gopalakrishnan et al. (2018); Sharma et al. (2019). Some preliminary results lead us to consider the “Block” lifting which considers a piecewise constant interpolation and corresponds to the one also used in Moon et al. (2019). Alternative piecewise linear or randomised orderings were also tried, but found not to be appreciably better to justify the added complexity. As we show for the example in Figure 6, this interpolation lifting divides an image in disjoint blocks via a coarse grid and associates to each of the blocks the same value of a parameter in . We characterise the lifting with the following conditions
(10)  
(11) 
Since may still be very high (usually ), for each level we apply domain subsampling and consider . We order the blocks according to the variance of mean intensity among neighbouring blocks, in contrast to the variance within each block which was suggested in Chen et al. (2017). Consequently, at each level the adversarial example is found by solving the following iterative problem
(12)  
where .
In its simplest formulation, hierarchical lifting struggles with the pixelwise interval constraint, . To address this we allow the entries in to exceed the interval and then reproject the pixelwise entries into the interval.
3.2 Algorithm pseudocode
4 Comparison of Derivative Free Methods
We compare the performance of our BOBYQA based algorithm to GenAttack Alzantot et al. (2019), combinatorial attacks COMBI Moon et al. (2019) and SQUARE Andriushchenko et al. (2019). The performance is measured by considering the distribution of queries needed to successfully find adversaries to different networks trained on three standard datasets: MNIST Lecun et al. (1998), CIFAR10 Krizhevsky (2009), and ImageNet Deng et al. (2009).
4.1 Parameter Setup for Algorithms
Our experiments rely for GenAttack Alzantot et al. (2019), COMBI Moon et al. (2019), and SQUARE Andriushchenko et al. (2019) on publicly available implementations
For the proposed algorithm based on BOBYQA, we tuned three main parameters: the dimension of the initial set , the batch dimension , and the trust region radius.

Batch Dimension Figure 7 shows the loss value averaged over 20 images for attacks to NNs trained on CIFAR10, and ImageNet datsets when different batch dimensions are chosen. The average objective loss as a function of network queries is largely insensitive to the batch sizes, but with modest differences for the larger ImageNet data set where was observed to require modestly fewer queries. For the remained of the simulations we use as a good tradeoff between faster model generation and good performances.

Initial Set Dimension Once a subdomain of dimension is chosen, the model (3) is initialised with a set of samples on which the interpolation conditions (4) are imposed. There are two main choices for the dimension of the set: either , thus computing and c with the interpolation and leaving M always null and thus having a linear model, or which allows us to initialise , and the diagonal of M, hence obtaining a quadratic model. The results in Figure 8 show that at each iteration of the domain subsampling the quadratic method performs as well as a linear method, however it requires more queries to initialise the model. Thus we consider the linear model with
^{4} . 
Trust Region Radius Once the model for the optimisation is built, the step of the optimisation is bounded by the trust region radius. We have selected the beginning radius to be one third of the whole space in which the perturbation lies. With this choice of radius we usually reach within 5 steps a corner of the boundary, and the further iterates remain effectively stationary.
For the hierarchical lifting approach we consider an initial subdomain of dimension , as this is the biggest grid that we can optimise over with a batch . After considering , we make use of and do not consider further levels.
4.2 Dataset and Neural Network Specifications
Experiments on each dataset are performed with one of the best performing NN architectures as described below
Mnist/cifar10
MNIST and CIFAR10 are two datasets with images divided between 10 classes and of dimension 28x28x1 and 32x32x3 respectively. On them we apply the net introduced in Chen et al. (2017) which is structured in succession by: 2 Conv layers with ReLu activation followed by a maxpooling layer. This process is repeated twice and then two dense layers with Relu activation are applied. Finally a softmax layer generates the output vector. For each dataset, we train the same architecture in two different ways obtaining separate nets. One is obtained by optimising the accuracy of the net on raw unperturbed images, while the other is trained with the application of the distillation defence by Papernot et al. (2016).
To generate a comprehensive distribution for the queries at each energy budget, for both the two trained nets and 10 images per class, we attempt to misclassify an image targeting all of the 9 remaining classes; this way we generate a total of 900 perturbations per energy budget. For these two datasets the images are of relative low dimension and we do not apply the hierarchical approach.
ImageNet
This is a dataset of millions of images with a dimension of 299x299x3 divided between 1000 classes. For this dataset we consider the Inceptionv3 net Szegedy et al. (2016) trained with and without the adversarial defence proposed in Kurakin et al. (2016)
4.3 Experimental Results
In Figure 9 we present the cumulative fraction of images misclassified (abridged by CDF for cumulative distribution function) as a function of the number of queries to the NN for different perturbation energies . The pixels are normalised to be in the interval , hence, would imply that any pixel is allowed to change of the total intensity range from its initial value. By illustrating the CDFs we easily see which method has been able to misclassify the largest fraction of images in the given testset for a fixed number of queries to the NN. It can be observed that the proposed BOBYQA based approach achieves stateoftheart results when the perturbation bound of decreases. This behaviour is consistent across all of the considered datasets (MNIST, CIFAR10, and ImageNet); however, the energy at which the BOBYQA algorithm performs the best, varies in each case.
In the experiments we also considered nets trained with defence methods, distillation Papernot et al. (2016) for MNIST and CIFAR10 datasets while adversarial training Kurakin et al. (2016) for ImageNet, and the results can be identified in Figure 9 by the dashed lines. Similar to the previous case, we observe that the proposed BOBYQA based algorithm performs the best when the energy perturbation decreases. Moreover, the BOBYQA based algorithm seems to be the least affected in its performance when the any defence is used; for example, at 0.01 and 15,000 queries, the defence reduces the CDF of COMBI by 0.078 compared to 0.051 for BOBYQA. This further supports the idea that for more challenging scenarios modelbased approaches are preferable as compared to modelfree counterparts.
We associate the counterintuitive improvement of the CDF in the MNIST and ImageNet with high perturbation energies cases to the distillation and the adversarial training being focused primarily on low energy perturbations. For ImageNet, nonmodelbased algorithms use different hierarchical approaches which we expect leads in part to the superior performance of COMBI in Fig. 9 panels (g)(i).
5 Discussion and Conclusion
We have introduced BOBYQA, a method to search adversarial examples based on a modelbased DFO algorithm and have conducted some experiments to understand how it compares to existing GenAttack Alzantot et al. (2019), COMBI Moon et al. (2019), and SQUARE Andriushchenko et al. (2019) attack, when targeted blackbox adversarial examples are searched with the fewest queries to a neural net.
Following the results of the experiments that we presented above, the method with which generating the adversarial example should be chosen according to which setting the adversary is considering. When the perturbation energy is high, one should choose either COMBI if the input is highdimensional or SQUARE if the input is lowdimensional. On the other hand, a modelbased approach like BOBYQA should be considered as soon as the complexity of the setting increases, e.g. the maximum perturbation energy is reduced or the net is adversarially trained.
With the BOBYQA attack algorithm we have introduced a different approach for the generation of targeted adversarial examples in a blackbox setting with the aim of exploring what advantages are achieved by considering modelbased DFO algorithms. We did not focus on presenting an algorithm which is in absolute the most efficient; primarily because our algorithm has several aspects in which to be improved. The BOBYQA attack is limited by the implementation of pyBOBYQA Cartis et al. (2019) since the elementwise constraints do not allow the consideration of more sophisticated liftings which leverage on compressed sensing, to name one of the many possible variations.
In conclusion, the results in this paper support how sophisticated misclassification methods are preferable in challenging settings. As a consequence, variations on our modelbased algorithms should be considered in the future as a tool to establish the effectiveness of newly presented adversarial defence techniques.
Acknowledgements
This publication is based on work supported by the EPSRC Centre for Doctoral Training in Industrially Focused Mathematical Modelling (EP/L015803/1) in collaboration with New Rock Capital Management.
Footnotes
 BOBYQA was selected among the numerous types of modelbased DFO algorithms due to its efficiency observed for other similar problems requiring few model samples as in climate modelling Tett et al. (2013)
 https://github.com/giughi/AModelBased
DerivativeFreeApproachtoBlackBox
AdversarialExamplesBOBYQA  GenAttack: https://github.com/nesl/adversarial_genattack
COMBI: https://github.com/snumllab/parsimoniousblackboxattack
SQUARE: https://github.com/maxandr/squareattack  The Constraint Optimisation by Linear Approximation (COBYLA), a linear based model DFO algorithm, was introduced before BOBYQA Powell (2007); however, COBYLA considers different constraints on the norm of the variable. Because of this and the possibility to extend the method to quadratic models, we name our algorithm after BOBYQA.
 For the nonadversarially trained net we considered the one available at http://jaina.cs.ucdavis.edu/datasets/adv/imagenet/inception_v3_2016_08_28_frozen.tar.gz, while for the weights of the adversarially trained net we relied on https://github.com/tensorflow/models/tree/master/research/adv_imagenet_models.
References
 GenAttack: practical blackbox attacks with gradientfree optimization. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pp. 1111–1119. External Links: Document Cited by: A ModelBased DerivativeFree Approach to BlackBox Adversarial Examples: BOBYQA, Figure 1, item Adversary’s Knowledge, §1, §1, §2, §3, §4.1, §4, §5.
 Square attack: a queryefficient blackbox adversarial attack via random search. arXiv preprint arXiv:1912.00049. Cited by: A ModelBased DerivativeFree Approach to BlackBox Adversarial Examples: BOBYQA, Figure 1, §1, §1, §3, §4.1, §4, §5.
 Towards evaluating the robustness of neural networks. In Proceedings of the IEEE Symposium on Security and Privacy (SP), pp. 39–57. External Links: Document Cited by: item Adversary’s Knowledge, §2.
 Improving the flexibility and robustness of modelbased derivativefree optimization solvers. ACM Trans. Math. Softw. 45 (3). External Links: Document Cited by: §3.2, §5.
 EAD: elasticnet attacks to deep neural networks via adversarial examples. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10–17. Cited by: item Adversary’s Knowledge.
 ZOO: zeroth order optimization based blackbox attacks to deep neural networks without training substitute models. In Proceedings of the ACM Workshop on Artificial Intelligence and Security (AISec), pp. 15–26. External Links: Document Cited by: item Adversary’s Knowledge, §1, §1, §2, §3.1, §3.1, §3, §4.2.
 Introduction to derivativefree optimization. Vol. 8, SIAM. Cited by: §3.
 Adversarial classification. In Proceedings of the ACM International conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 99–108. External Links: Document Cited by: §1.
 ImageNet: a largescale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. External Links: Document Cited by: §4.
 Stochastic activation pruning for robust adversarial defense. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §1.
 Robust physicalworld attacks on deep learning visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1625–1634. External Links: Document Cited by: §1.
 Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: item Adversary’s Knowledge, §1.
 Toward robust neural networks via sparsification. arXiv preprint arXiv:1810.10625. Cited by: §3.1.
 Low frequency adversarial perturbation. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), Cited by: §3.1.
 Delving deep into rectifiers: surpassing humanlevel performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034. External Links: ISBN 9781467383912, Document Cited by: §1.
 Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Processing Magazine 29 (6), pp. 82–97. External Links: Document Cited by: §1.
 Blackbox adversarial attacks with limited queries and information. In Proceedings of the International Conference on Machine Learning (ICML), pp. 2137–2146. Cited by: item Adversary’s Knowledge.
 Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §4.
 Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236. Cited by: Figure 9, §4.2, §4.3.
 Adversarial examples in the physical world. In Proceedings of the International Conference on Learning Representations (ICLR), Workshop Track, Cited by: §1.
 Derivativefree optimization methods. Acta Numerica 28, pp. 287–404. External Links: Document Cited by: §3.
 Gradientbased learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. External Links: Document Cited by: §4.
 Fake news detection on social media using geometric deep learning. arXiv preprint arXiv:1902.06673. Cited by: §1.
 Parsimonious blackbox adversarial attacks via efficient combinatorial optimization. In Proceedings of the International Conference on Machine Learning (ICML), pp. 4636–4645. Cited by: A ModelBased DerivativeFree Approach to BlackBox Adversarial Examples: BOBYQA, Figure 1, §1, §1, §3.1, §3, §4.1, §4, §5.
 Simple blackbox adversarial attacks on deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1310–1318. External Links: Document Cited by: item Adversary’s Knowledge.
 Numerical optimization. SpringerVerlag New York. External Links: Document, ISBN 9780387303031 Cited by: §3.
 Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the IEEE Symposium on Security and Privacy (SP), pp. 582–597. External Links: Document Cited by: Figure 1, Figure 9, §4.2, §4.3.
 Practical blackbox attacks against machine learning. In Proceedings of the ACM on Asia Conference on Computer and Communications Security (ASIA CCS), pp. 506–519. External Links: Document Cited by: §1.
 A view of algorithms for optimization without derivatives. Mathematics TodayBulletin of the Institute of Mathematics and its Applications 43 (5), pp. 170–174. Cited by: footnote 4.
 The bobyqa algorithm for bound constrained optimization without derivatives. Technical report Technical Report DAMTP 2009/NA06, University of Cambridge. Cited by: A ModelBased DerivativeFree Approach to BlackBox Adversarial Examples: BOBYQA, §1, §3.
 On the effectiveness of low frequency perturbations. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 3389–3396. External Links: Document Cited by: §3.1.
 DARTS: deceiving autonomous cars with toxic signs. arXiv preprint arXiv:1802.06430. Cited by: §1.
 Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826. External Links: Document Cited by: §4.2.
 Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §1.
 Can topofatmosphere radiation measurements constrain climate predictions? part i: tuning. Journal of Climate 26 (23), pp. 9348–9366. External Links: Document Cited by: footnote 1.
 Autozoom: autoencoderbased zeroth order optimization method for attacking blackbox neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence: Special Technical Track: AI for Social Impact, External Links: Document Cited by: §1.
 Protecting neural networks with hierarchical random switching: towards better robustnessaccuracy tradeoff for stochastic defenses. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI, Cited by: §1.
 Adversarial examples: attacks and defenses for deep learning. IEEE Transactions on Neural Networks and Learning Systems 30 (9), pp. 2805–2824. External Links: Document Cited by: §1, §1.