Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics
Abstract
Deep learning has greatly improved visual recognition in recent years. However, recent research has shown that there exist many adversarial examples that can negatively impact the performance of such an architecture. This paper focuses on detecting those adversarial examples by analyzing whether they come from the same distribution as the normal examples. Instead of directly training a deep neural network to detect adversarials, a much simpler approach was proposed based on statistics on outputs from convolutional layers. A cascade classifier was designed to efficiently detect adversarials. Furthermore, trained from one particular adversarial generating mechanism, the resulting classifier can successfully detect adversarials from a completely different mechanism as well. The resulting classifier is nonsubdifferentiable, hence creates a difficulty for adversaries to attack by using the gradient of the classifier. After detecting adversarial examples, we show that many of them can be recovered by simply performing a small average filter on the image. Those findings should lead to more insights about the classification mechanisms in deep convolutional neural networks.
1 Introduction
Recent advances in deep learning have greatly improved the capability to recognize visual objects [13, 26, 7]. Stateoftheart neural networks perform better than human on difficult, largescale image classification tasks. However, an interesting discovery has been that those networks, albeit resistant to overfitting, would have completely failed if some of the pixels in the image were perturbed via an adversarial optimization algorithm [28, 4] . An image indistinguishable from the original for a human observer could lead to significantly different results from a deep network(Fig. 1).
Those adversarial examples are dangerous if a deep network is utilized in any crucial real application, be it autonomous driving, robotics, or any automatic identification (face, iris, speech, etc.). If the result of the network can be hacked at the will of a hacker, wrong authentications and other devastating effects would be unavoidable. Therefore, there are ample reasons to believe that it is important to identify whether an example comes from a normal or an adversarial distribution. A reliable procedure can prevent robots from behaving in undesirable manners because of the false perceptions it made about the environment.
The understanding of whether an example belongs to the training distribution has deep roots in statistical machine learning. The i.i.d. assumption was commonly used in learning theory, so that the testing examples were assumed to be drawn independently from the same distribution of the training examples. This is because machine learning is only good at performing interpolation, where some training examples surround a testing example. Extrapolation is known to be difficult, since it is extremely difficult to estimate data labels or statistics if the data is extremely different from any known or learned observations. Many current approaches deal with adversarial examples by adding them back to the training set and retrain. However in their experiments, new adversarials can almost always be found from the retrained classifier. This is because that the space of extrapolation is significantly larger than the area a machine learning algorithm can interpolate, and the ways to find vulnerabilities of a deep learning system are almost endless.
A more conservative approach is to refrain from making a prediction if the system does not feel comfortable about it. Such an approach seeks to build a wall to fence all testing examples in the extrapolation area out of the predictor, and only predict in the small interpolation area. Work such as [16] provides basic theoretical frameworks of classification with an abstain option.
Although these concepts are wellknown, the difficulties lie in the highdimensional spaces that are routinely used in machine learning and especially deep learning. Is it even possible to define interpolation vs. extrapolation in a dimensional or dimensional space? It looks like almost everything is extrapolation since the data is inherently sparse in such a highdimensional space [9, 6], a phenomenon wellknown as the curse of dimensionality. The enforcement of the i.i.d. assumption seems impossible in such a highdimensional space, because the inverse problem of estimating the joint distribution requires an exponential number of examples to be solved efficiently. Some recent work on generative adversarial networks proposes using a deep network to train this discriminative classifier [3, 22], where a generative approach is required to generate those samples, but it is largely confined to unsupervised settings and may not be applicable for every domain convolutional networks (CNNs) have been applied to.
In this work we propose a discriminative approach to identify adversarial examples, which trains on simple features and can approach good accuracy with limited training examples. The main difference between our approach and previous outlier detection/adversarial detection algorithms (e.g. [2]) is that their approaches usually treat deep learning as a black box and only works at the final output layer, while we believe that the learned filters in the intermediate layers efficiently reduce the dimensionality and are useful for detecting adversarial examples. We make a number of empirical visualizations that show how the adversarial examples change the prediction of a deep network. From those intuitions, we extract simple statistics from convolutional filter outputs of various layers in the CNN. A cascade classifier is proposed that utilizes features from many layers to discriminate between normal and adversarial examples.
Experiments show that our features from convolutional filter output statistics can separate between normal and adversarial examples very well. Trained with one particular adversarial generation method, it is robust enough to generalize to adversarials produced from another generation approach [20] without any special adaptation or additional training. Those confidence estimates may improve the safety of applying these deep networks, and hopefully provide insights for further research on selfaware learning. As a simple extension, the results from visualizations of the features prompted us to perform an average filter on corrupted images, and found out that many correct predictions can be recovered from this simple filtering.
2 Deep Convolutional Neural Networks
A deep convolutional neural network consists of many convolutional layers which are connected to spatially/temporally adjacent nodes in the next layer:
(1) 
where is the input features at layer , are filters that could be much smaller than the size of (e.g. , ), is the convolution operator, and is a nonlinear transformation function such as the rectified linear unit (ReLU) . Other commonly used layers in a CNN include maxpooling layers, or other normalization layers [13] such as batch normalization layers [10]. Most deep networks adopt similar principles while adding more structural complexity in the system such as more layers and smaller filters in each layer [26], multilayered network within each layer [27], residual network [7], etc. A convolutional neural network makes sense in structured data because it naturally exploits the locality structure in data. In an image, pixels that are located close to each other are naturally more correlated than pixels that are far away [17]. The same holds for temporal data (video, speech) where objects (frames, utterances) that are temporally close can be assumed to be more correlated.
3 Understanding the Trained Deep Classifier Under Adversarial Optimization
3.1 Adversarial Optimization
The famous result that deep networks can be broken easily [28] is an important motivation of this work. The idea is to start from an existing example (image) and optimize to obtain an example that will be classified to another category while being close to the original example. Namely, the following optimization problem is solved:
(2) 
where is a known example and is an arbitrary category label, is the input dimensionality. is a parameter that can be tuned for trading off between proximity to the original example and the classification loss on the other category . It has been shown, to the astound of many, that one can choose an with very small norm while completely change the output of the algorithm (e.g. Fig. 1), this can even be done universally for almost all networks, datasets and categories [28, 4]. Besides, adversarials trained from one network may even fool a related one trained from the same dataset [18]. This has led many people to question whether deep networks are really learning the “proper” rules for classifying those images.
3.2 Adversarial Behavior
In order to gain a deeper understanding of the behavior of a deep network and illustrate the difference between adversarial and normal example distributions, we utilize spectral analysis. As a starting point, we perform principal component analysis (PCA) [11] at the 14th layer of a VGG network trained on the ImageNet dataset (the first fullyconnected layer). The rationale behind using PCA is that each deep learning layer is a nonlinear activation function on a linear transformation, hence a large part of the learning process lies within the linear transformation, for which PCA is a standard tool to analyze.
A linear PCA is performed on the entire collection of images from the ImageNet validation set, as well as adversarials collected using the approach in (2), starting from random images in the collection. The result shows very interesting findings (Fig. 2) and sheds more light on the internal mechanics of those adversarial examples. In Fig. 2(a), we show the PCA projection onto the first two eigenvectors. This cannot separate normal and adversarial examples, as one could possibly imagine. The adversarial examples seem to exactly belong to the same distribution as normal ones. However, it does seem that the adversarial examples reside mostly in the center while the normal examples occupy a bigger chunk of space.
Interestingly, as we move to the tail of the PCA projection space, the picture starts to change significantly. In Fig. 2(b), we can see that there are a significant amount of adversarial examples that has extremely large values w.r.t. to the normal examples in the tail of the distribution. We chose to print the projection on the th and th eigenvector, but similar distributions can be found all over the tail. As one can see, at such a far end on the tail, the projections of normal examples are very similar to random samples under a Gaussian distribution. An explanation for that could be that under these “uninformative” directions, most of the weighted features are nearly independent w.r.t. each other, hence the distribution of their sum is similar to Gaussian, according to the central limit theorem^{1}^{1}1Note this is without a ReLU transformation. ReLU would destroy the negative part of the data distribution so that it no longer looks like a Gaussian. However, some tail effects can be observed even in the distribution after ReLU.. However, although normal examples behave similarly to a Gaussian, some adversarial examples are having projections with a deviation as large as or times the standard deviation, which are extremely unlikely to occur under a Gaussian distribution.
(a)  (b)  (c)  (d) 
Fig. 2(c) and Fig. 2(d) show that there are two distinct phenomena:

The extremal values and standard deviations on the projections onto the first eigenvectors are decidedly lower in adversarial examples than in normal ones.

The extremal values and standard deviations on the projections onto the last eigenvectors are decidedly higher in the adversarial examples than the normal ones.
It is interesting to reflect about the causes and consequences of those properties. One deciding property is that there is a strong regularization effect in adversarial examples on almost all the informative directions. Hence, the predictions in adversarial examples are lower than those in normal examples, rather than the confidence values may have indicated (Fig. 1). In Fig. 3, we show the number of categories with a prediction higher than a threshold, before the final softmax transformation
(3) 
that converts raw predictions into probabilities. The result shows that normal examples have on average one category with a raw prediction value more than , however adversarial examples have only category with raw predictions more than . The reason that those adversarial examples appear more confident after softmax is because that the predictions on all the other categories are regularized even more. Hence the normalization component of softmax has decided that the single prediction, although much less strong, should be assigned a probability of more than . We note that this issue was also pointed out by [2] in a different manner and they proposed a solution in the OpenMax classifier, which we compare against in the experiments.
But besides that, it seems that such extremal and standard deviation statistics are evident features that could help discriminating normal and adversarial examples. Unfortunately, they only occur as a statistic from a large sample, as any single point in Fig. 2(a) looks similar to a single point in the normal distribution. We have tried to utilize the tail distributions (Fig. 2(b)) to create a classifier which easily achieved more than accuracy separating adversarials from normals, however we subsequently found out that since the tail almost do not contribute to the classification, knowing this defense, the adversarial example can deliberately remove their footprints on the tail distributions.
This leads us to think about an approach that would turn a single image into a distribution, so that we can use statistics as detectors for adversarial examples. An image is a distribution of pixels. Especially, the output of each filter from each convolutional layer is an image which could be treated as a distribution where the samples are the pixels. Therefore, in the following section we aim to build a classifier based on collecting statistics from such distributions.
4 Identifying Adversarial Examples
4.1 Feature Collection
Suppose the output at a convolutional layer is an tensor, where and represent the width and height of the image at that stage (smaller than original after maxpooling), and represents the number of convolutional filters. Such a tensor can be considered as a channel image where each pixel has a dimensional feature. We consider the feature on every pixel to be a random vector drawn from the distribution of convolutional pixel outputs, a dimensional distribution.
(a)  (b) 
The list of statistics we collect is:

Normalized PCA coefficients

Minimal and Maximal values

25th, 50th and 75th percentile values
on each of the dimensional features. Normalized PCA coefficients are collected via Algorithm 1. Extremal and percentile statistics are straightforward to understand.
The features we collect are nonsubdifferentiable, hence essentially preventing adversaries to use gradientbased attacks to counter the classifier. Although we are interested in a generative adversarial networktype adversary which would learn to avoid our detector, such adversaries would have to resort to derivativefree optimization methods, which currently do not scale to the size of a realistic image. The best derivativefree approach we have tried scales up to several hundreds of variables. The genetic algorithm in [20] scales better, but as we will soon show, their lowlevel feature statistics are so different from natural images, making them very easy to be detected, even without training on any data from their adversarial generation algorithm.
4.2 Classifier Cascade
[29] proposed a famous strategy for face detection by using a cascaded boosting classifier composed by a sequence of base classifiers. A cascade classifier is ideal when it is easy to identify many of the examples from a category but some important cases can be difficult. In Fig. 4, represents the classifier at each stage. is the input of the cascade classifier. The negatives in a cascade classifier from each stage will be outputted directly, while the positives will go to the next stage.
In our case, the normal category is much easier to detect than the adversarial category (see e.g. Fig. 6). In our initial experiments with VGGNet, we found that more than of normal examples can be determined from the first convolutional layer with precision. Therefore, we constructed a cascade classifier based on convolutional layers: the first stage works with features collected from the outputs of the first convolutional layer, the second with the second layer, etc. (Fig. 4). The base classifiers will not solely consider statistics from their own stage, instead, after one stage of training, the remaining positive examples will be concatenated to the corresponding features on the next stage.
The overall false positive rate of a stage cascade classifier can be represented as: , where is the false positive rate at each layer. And similarly the true positive rate can be represented in the same form: where is the true positive rate at each stage. In order to maximize recall, we maintain a high true positive rate and select a classification threshold which corresponds to a high true positive rate ( in AlexNet and in VGG).
5 Related Work
Szegedy et al. [28] proposes the adversarial optimization formulation in eq. (2). [4] proposes an explanation of the adversarial mechanism, and proposed a simpler adversarial optimization mechanism that only corrupts based on the signs of gradient of the network. The fact that such examples can be generated so easily with the gradient sign method shows that adversarial examples come from attacking the magnifying effect coming from the linearities in the network. [20] proposes another mechanism to generate adversarials using evolutionary optimization. The result of these do not resemble natural images but still can be classified by deep networks with high confidence(Fig. 5). [19] proposes another efficient approach. [23] proposes an approach to generate adversarials that match the convolutional filter outputs as well as perturbing the data. [25, 8] propose approaches to sample adversaries or minimax optimization for making learning more robust. While most of the work are done on standard benchmarks such as MNIST, CIFAR and ImageNet, [14] is an interesting work on projecting the adversaries in physical world.
Recently, there have been a lot of focus on training adversarial generation networks to create Generative Adversarial Networks (GANs) [3, 22, 32, 24]. These networks play a twoplayer game where a generator network aims to generate adversarials that will not be correctly classified by another discriminator network, and the goal is to generate images more and more similar to natural images. It has been shown that these networks generate images that resemble natural images. However, this generative approach is different from our goal, where we aim to create discriminative networks that discriminates from images that are already indistinguishable from natural images (e.g. Fig.1).
Mechanisms for detecting and countering adversarial examples have also been proposed [5, 21]. [18] proposes to use the foveation mechanism to alleviate adversarial examples when it is already known to be adversarial, but did not attempt to detect adversarials. The openset deep networks proposed by [2] seek to alleviate concerns from a softmax classification by creating an abstain option. The universum classifier [31] is similar but with more theoretical arguments.
Selfaware learning (classification with an abstain option) had been proposed in e.g. [12, 16]. It is relevant to robust learning (e.g. [15]), however robust learning usually seek to directly optimize the minimax loss under adversarial conditions, instead of outputting an abstain option. [1, 30] also focuses on classification with an abstain option.
6 Experiments
Our algorithm is tested on 2 approaches to generate adversaries. The main one is data generated using the LBFGS algorithm by [28]. We generated adversarials from a random subset of the ILSVRC2012 validation set (total of images). In order to test the outofsample generalization capability, we included another dataset, which includes EAadversarial images generated using the algorithm in [20]. These datasets are tested with 2 different network structures, VGG16 [26] and AlexNet [13]. All input images had been normalized and reshaped into for VGG16 model and for AlexNet. The MatConvNet toolbox is used for the deep networks. Features are extracted from the statistics of interest, mentioned in section 4.1, in the lower convolutional layers from the VGG16 network, and in all convolutional layers from AlexNet.
All the classifiers were trained with a subset of LBFGS adversarials and a randomly selected subset of normal images in the ILSVRC2012 dataset. Testing was done on a random heldout dataset with the other normal images and LBFGS adversarials for the LBFGS experiments, and on a randomly selected normal images and EAadversarials for the EA experiments. The training and testing sets do not overlap in any means: for all the testing LBFGS adversarials, the corresponding normal images were not seen in the training set. EA experiments are only conducted with AlexNet since we found out that EAadversarials did not succeed fooling the VGG16 network. We have also tried some other fast adversarial generation approaches such as [4], but they generally do not generate adversarials that fool classifiers as completely (with more than confidence) or as universally (on of the images) as the LBFGS algorithm.
Additional experiment results using DeepFool and on ResNet can be found in the supplementary material.
6.1 SingleLayer Results
The first experiments we did were by extracting statistics on each single convolutional layer output, instead of using the cascade classifier. From Tables 1 and 2, one can see that the statistics from a single layer have some discriminative power to distinguish normal examples from adversarials, but are not extremely effective. However, EAadversarials were much easier to distinguish, even though our classifier was trained only on LBFGS adversarials instead of EA ones (Table 3). We only need the first three convolutional layers to reach an overall classification accuracy. We believe the reason is that our features capture natural image statistics, and because EAadversarials look so unnatural, their statistics are vastly different than natural images (see Sec. 6.3 for more discussions).
Network Layer  2nd  3rd  4th 

Accuracy  
Network Layer  5th  6th  
Accuracy 
Network Layer  2nd  3rd  4th 

Accuracy  
Network Layer  5th  6th  7th 
Accuracy  
Network Layer  8th  9th  10th 
Accuracy 
Layer  2nd  3rd  4th 

Accuracy 
6.2 Experiment for LBFGSAdversarials Detection
Next we test the cascade classifier on both AlexNet and VGG16. The parameter is set to . On AlexNet, the average accuracy of the cascade classifier reaches over random trials, and the AUC (areaundercurve) metric is . We compared against the recently published OpenMax method [2]. To learn the Weibull distribution required for OpenMax, the EVT was applied on the same training set as the algorithm. Figure 6(a) shows the results, where we were able to outperform OpenMax by over in areaundercurve (AUC) and in terms of accuracy.
In VGG16, the results were even better. The accuracy of the classifier was on average over random trials. Fig.6(b) shows the ROC curve. We believe the fact that VGG has a lot more layers than AlexNet helps setting more constraints on the layer statistics, and is subsequently helpful for detecting adversarial examples.
Finally, the cascade classifier was tested on EAadversarials. We obtained more than accuracy with false positive rate, with a final accuracy of and AUC of (Fig.6(c)). In other words, our algorithm is rarely fooled by EAadversarials, even without training on them.
(a)  (b)  (c) 
6.3 Visualization of Statistics
Our experiment results show that EAadversarials are easy to detect with our detector. To gain more insight into this result, we made a few comparisons between the statistics of interest extracted from normal images, LBFGSadversarials and EAadversarials.
We visualized the average of the statistics that are used for the detection task from the first layer of the AlexNet on all its dimensions. As can be seen in Fig.7(a), the difference on the PCA projection statistics on extracted from EAadversarials and that of the normal images is very dramatic. Meanwhile, compared to the EAadversarials, the statistics from LBFGSadversarial have much less difference from the normal data and the difference does not change very much across different dimensions.
From Fig. 7(b), one can see that LBFGSadversarials have smaller extremal values than normal images. This might imply that the LBFGS optimization worked to diminish strong signals from the original image by introducing small pixel perturbations, and that helped our classifiers separating them from normal images. From Fig. 7(c), we see the EAadversarials evidently differ from normal images. Those results illustrate why EAadversarials are easier to detect. We suspect it would be easy to reach accuracy, had we actually trained on some EAadversarials. The capability to generalize to EAadversarials without training on them showed the general capability of our cascade classifiers to capture natural image statistics and distinguish natural images from unnatural ones.
(a)  (b)  (c) 
7 Discussions
7.1 SelfAware Learning with an Abstain Option
The framework of selfaware learning [16, 2, 31] considers the case where the learning algorithm has an abstain option of saying “I don’t know”, instead of always making an actual prediction. We define a framework that is slightly different than [16], avoiding the requirement in some frameworks of never making a mistake.
We assume that the training input is drawn i.i.d. from a distribution , where is the input and is the output. Assume that the testing input is drawn from a mixture distribution between and :
(4) 
, where is an unknown mixture weight, and is an adversarial distribution. Assume that we have a classifier that includes a function , and a boolean strategy between predict and abstain that can be chosen for each individual . Assume that the expected error from our classifier on the adversarial distribution is (which could be assumed, if no other prior is present, as the random guessing error of for a class classification problem). Further assume that abstaining always incur a fixed cost . As long as , abstaining would be better than predicting on the example drawn from the adversarial distribution, however, should be set sufficiently large so that the classifier would still make predictions when confident, instead of abstaining everything.
For each testing input, the testing of the selfaware classifier is then trying to optimize where
(5) 
hence the classifier needs to select between making a prediction using its function and risk paying versus abstaining. It is easy to derive the optimal strategy:
otherwise  (7) 
Our approach can be seen as estimating in this framework. Experiments about the effect of such selfaware learning is shown in the supplementary material. We eagerly hope to apply it in realistic applications in future work.
7.2 Image Recovery
Insights from [4] indicate that the adversarial mechanism is very specifically attacking vulnerable gradients starting from the first convolutional layer. Insights from the previous experiments also suggest that LBFGSadversarials work to diminish filter responses from the first convolutional layer. Therefore a natural idea would be to destroy the adversarial effects in the first convolutional layer to try to recover the original image. We tried a very simple approach: applying a small (e.g. ) average filter on the adversarial image before using the CNN to classify it. The positive and negative adverse gradients will average out in this approach, and make the masked activations from the normal images more prominent. In Table 4 we illustrate such recovery results: after using a average filter on identified adversarial examples, the classification accuracy improved from almost to , showcasing the effectiveness of this simple average filter.
Approach  Top5 Accuracy 

(Recovered Images)  
Original Image (Noncorrupted)  
Average Filter  
Average Filter  
Foveation (Object Crop MP) [18] 
Those results show that we can both detect and recover from adversarial examples with high accuracy. But the main reason we performed this (overly simplistic) experiment is to show how simple it might be to cancel out some adversarial perturbations. Importantly, this result indicates that current deep convolutional networks are too locally focused: these are corruptions that can be cancelled out by a simple average filter, however they can adversely impact the entire result of the deep network. For human with a large receptive field, they will not even care about what happens within a area. Therefore, we believe that future deep learning approaches should focus on enlarging the receptive field in order to reduce the chance of being fooled by adversarial examples. Another potential direction is to research classification approaches that do not require a softmaxtype normalization, in order to avoid regularizing attacks such as the ones used in the adversarial optimization in (2).
8 Conclusion
This paper proposes an approach that detects adversarial examples using simple statistics on convolutional layer outputs. A cascade classifier was designed based on simple statistics on filter outputs from each layer. And it was capable of detecting more than of the adversarial examples. Experiments showed that our cascade classifier significantly outperforms stateoftheart on detecting adversarial examples. Experiment also showed transfer learning capabilities of our classifier, since the classifier we trained with LBFGS adversarials are capable of detecting EAadversarials as well. Insights drawn from these experiments lead us to perform simple average filter to corrupted images, which successfully recovered most of them. In the future, we would like to explore GANtype generative adversarial networks from the current results, with multiple rounds of adversarial detection and counterdetection.
Acknowledgements
This paper was supported by Future of Life grants 2015143880 and 2016158701.
References
 [1] A. Balsubramani. Learning to abstain from binary prediction. arXiv preprint arXiv:1602.08151, 2016.
 [2] A. Bendale and T. E. Boult. Towards open set deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
 [3] I. Goodfellow, J. PougetAbadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
 [4] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
 [5] S. Gu and L. Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068, 2014.
 [6] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. SpringerVerlag, New York, 2001.
 [7] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
 [8] R. Huang, B. Xu, D. Schuurmans, and C. Szepesvári. Learning with a strong adversary. In International Conference on Learning Representations, 2016.
 [9] P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604–613, 1998.
 [10] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
 [11] I. Jolliffe. Principle Component Analysis. SpringerVerlag, 1986.
 [12] R. Kleinberg, A. NiculescuMizil, and Y. Sharma. Regret bounds for sleeping experts and bandits. Machine learning, 80(23):245–272, 2010.
 [13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012.
 [14] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
 [15] G. R. Lanckriet, L. E. Ghaoui, C. Bhattacharyya, and M. I. Jordan. A robust minimax approach to classification. Journal of Machine Learning Research, 3:555–582, 2003.
 [16] L. Li, M. L. Littman, T. J. Walsh, and A. L. Strehl. Knows what it knows: a framework for selfaware learning. Machine learning, 82(3):399–443, 2011.
 [17] X. Li, F. Li, X. Fern, and R. Raich. Filter shaping for convolutional networks. In International Conference on Learning Representations, 2017.
 [18] Y. Luo, X. Boix, G. Roig, T. A. Poggio, and Q. Zhao. Foveationbased mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292v3, 2016.
 [19] S. MoosaviDezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. CoRR, abs/1511.04599, 2015.
 [20] A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
 [21] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. arXiv preprint arXiv:1511.04508, 2015.
 [22] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
 [23] S. Sabour, Y. Cao, F. Faghri, and D. J. Fleet. Adversarial manipulation of deep representations. In International Conference on Learning Representations, 2016.
 [24] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. arXiv preprint arXiv:1606.03498, 2016.
 [25] U. Shaham, Y. Yamada, and S. Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. arXiv preprint arXiv:1511.05432, 2015.
 [26] K. Simonyan and A. Zisserman. Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556, 2014.
 [27] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv:1409.4842, 2014.
 [28] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
 [29] P. Viola and M. J. Jones. Robust realtime face detection. International journal of computer vision, 57(2):137–154, 2004.
 [30] Y. Wiener and R. ElYaniv. Agnostic selective classification. In Advances in Neural Information Processing Systems, pages 1665–1673, 2011.
 [31] X. Zhang and Y. LeCun. Universum prescription: Regularization using unlabeled data. In AAAI Conference on Artificial Intelligence, 2017.
 [32] J. Zhao, M. Mathieu, and Y. LeCun. Energybased generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
Supplementary Material
Appendix A Results on DeepFool
For this experiment we used 5000 adversarial images generated with the DeepFool algorithm. We collected a training set of images using the adversarials, with an equal number of real images drawn from the ILSVRC2012 validation set. For the Deep Fool adversarials we used the implementation given in the Foolbox algorithm library. Given this data, we chose to use the ResNet 50 architecture as the backbone CNN for our experiments. Because we use ImageNet data, we preprocessed our training set accordingly: all images were reshaped to 224x224x3, the channels were modified to BGR ordering, and the channelwise mean was subtracted from each sample. We then performed a forward pass as usual on all images, performing the classifier cascade at each layer. Given the output of each convolutional layer, we extracted features that would characterize images as being either from a real or adversarial distribution. For each output we extracted PCA coefficients, extremal values, and values within the 25th, 50th and 75th percentile to form a new training sample. We then used an SVM to learn the real statistics from the adversarials. The real examples were given a ground truth label of 1, and the adversarials were given a label of 0. We performed a parameter search over kernel type and C value. In all our experiments a C of 0.005 and a linear kernel performed the best. We then tested on 2000 real images and 2000 DeepFool adversarials. In our experiments with ResNet, we only performed the cascade for the first three layers before we found nearly all of the adversarial images.
The result can be seen in Fig. 1. The algorithm maintained more than AUC, showing that DeepFool did not fundamentally change the type of adversarials.
Appendix B Results on SelfAware Learning
We implemented self aware learning using ResNet 50, the goal being to use the softmax probabilities to learn parameters that would cause the network to more intelligently classify inputs. The network should classify an image if it was sure the image was real, or abstain if either the network was not sufficiently confident, or if the image was adversarial, as described in Sec. 7.1. To test the presented algorithm, we use real images drawn from ILSVRC2012 validation set, and adversarial images from the testing set of the previous experiment, generated using the DeepFool algorithm. We tested the self aware learning algorithm with a high . This worked well enough that the network chose to abstain or classify, rather than incur a high penalty for guessing incorrectly. We observed that for each testing image, our estimation of the source distribution resulted in between 2 and 8. We then varied between these values to see if there was a threshold at which we could abstain from all adversarials, retaining predictions for only real examples. We were also interested in thresholds that maximized the true positive rate (prediction of real examples) while abstaining from as many adversarials as possible. We found the lower thresholds resulted in the abstaining from predicting on all adversarials, but it also abstained from many (but not all) real examples. Higher thresholds resulted in many more real predictions retained, but some also some adversarials made it through. High thresholds would finally result in the network not abstaining at all.
The results can be seen in Fig. 2. It can be seen that besides abstaining adversarial examples, the system also abstains from predicting on some normal examples that the classifier is not confident on. Hence, with a high abstain ratio the prediction accuracy on normal examples is also higher.
Appendix C Images Classified Correctly and Incorrectly
In this section we show some images classified correctly and incorrectly from the algorithm. Unfortunately we are not quite able to observe any particular visible trends, maybe due to the subtlety of adversarial images.
(a)  (b)  (c)  (d) 
(a)  (b)  (c)  (d) 
(a)  (b)  (c)  (d) 
(a)  (b)  (c)  (d) 