Are Odds Really Odd?
Bypassing Statistical Detection of Adversarial Examples
Abstract
Deep learning classifiers are known to be vulnerable to adversarial examples. A recent paper presented at ICML 2019 proposed a statistical test detection method based on the observation that logits of noisy adversarial examples are biased toward the true class. The method is evaluated on CIFAR10 dataset and is shown to achieve true positive rate (TPR) at only false positive rate (FPR).
In this paper, we first develop a classifierbased adaptation of the statistical test method and show that it improves the detection performance. We then propose Logit Mimicry Attack method to generate adversarial examples such that their logits mimic those of benign images. We show that our attack bypasses both statistical test and classifierbased methods, reducing their TPR to less than and , respectively, even at FPR. We finally show that a classifierbased detector that is trained with logits of mimicry adversarial examples can be evaded by an adaptive attacker that specifically targets the detector. Furthermore, even a detector that is iteratively trained to defend against adaptive attacker cannot be made robust, indicating that statistics of logits cannot be used to detect adversarial examples.
1 Introduction
Adversarial examples are inputs to machine learning models that an attacker intentionally designs to cause the model to make a mistake [1]. One particular method for generating adversarial examples for image classifiers is adding small perturbations to benign inputs such that the modified image is misclassified by the model, but a human observer perceives the original content [2, 3].
Several methods have been proposed for developing robust algorithms that classify adversarily perturbed examples into their groundtruth label, of which adversarial training is shown to be an effective approach [4]. Several other defenses have been broken with adaptive iterative attacks [5, 6, 7, 8, 9, 10]. A different approach for defending against adversarial examples is to detect whether an input is adversarially manipulated. Several such detection methods have been proposed, but later shown to be ineffective [11, 12, 13].
A recent paper presented at ICML 2019 proposed a detection method based on a statistical test that measures how logits change under noise [14]. The authors posited that robustness of logits are different, depending on whether the input is naturally generated or adversarially manipulated. Figure 1 visualizes this observation by measuring the average true probability across random and adversarial directions. The conelike structure, referred to as adversarial cone in [14], indicates that the adversarial sample is “surrounded” by the true class. The authors then proposed to add noise to inputs as an approach to partially undo the effect of adversarial perturbations and, thus, detect and correctly classify adversarial examples. The method is evaluated on CIFAR10 dataset and is shown to achieve true positive rate (TPR) at only false positive rate (FPR).
Unlike adversarial training method, several proposals for robust classifiers or detectors rely on “subtle” properties of models or benign or adversarial examples that are not “tied” to how the classification works, i.e., the model is not required to satisfy those properties to achieve good classification performance. Such defenses can be bypassed by Mimicry Attack, in which the attacker generates adversarial examples to not only fool the classifier, but also mimic the behavior of benign examples, where the specific behavior is derived from the mechanism that distinguishes between benign and adversarial examples. In [14], the authors observed that logits of adversarial examples are not robust to noise. In this paper, we show that this is a subtle property that is not fundamental to adversarial examples or deep learning classifiers, i.e., adversarial examples can be crafted such that they are misclassified by the model and their logits are indistinguishable from logits of benign examples. Our contributions are summarized in the following.

We first propose a classifierbased adaptation of the statistical test method. The proposed classifier takes logits of clean and noisy images as input and detects whether the image is adversarial. We show that such a classifier is able to identify anomalies in adversarial logits beyond their directionality toward the true class. Specifically, it improves the detection performance of the statistical test method against CW attack and succeeds even in small noise regimes where the test fails.

We then propose Logit Mimicry Attack (LMA), where the attacker collects logit profiles of clean and noisy benign images and crafts adversarial examples such that their logits are similar to those of benign images. We perform experiments on ResNet56 network and CIFAR10 dataset and show that our attack bypasses both statistical test and classifierbased methods, reducing the TPR to less than and , respectively, even at FPR of . Figure 1 visualizes the changes of average true probability across random and adversarial directions and shows that mimicry adversarial examples do not display the cone structure of the examples obtained from CW attack and are nearly as robust to noise as benign samples.

We then propose a mimicry detector, a classifier that is trained with logits of mimicry adversarial examples. Such a classifier improves the detection performance against LMA and achieves TPR of at FPR. We, however, show that an adaptive attacker that generates adversarial examples that evade both classifier and detector can successfully bypass the mimicry classifier, reducing the TPR to at FPR of .

We finally consider a detector that is iteratively trained against adaptive attack, i.e., at each iteration, adversarial examples are generated to bypass the detector and then the detector is trained to classify them as adversarial. Once training is done, the final detector is tested against the adaptive attack. We show that such a detector only achieves the TPR of at FPR. The results indicate that logits of benign and adversarial exmaples are not fundamentally distinguishable, i.e., new adversarial examples can be always generated that evade both the network and any detector.
The code for mimicry attack is available at https://github.com/HosseinHosseini/MimicryAttack.
2 Preliminaries
In this section, we provide the notations used in this paper and review the projected gradient descent method for generating adversarial examples.
2.1 Notations
Let be a function that takes an image , where is the number of pixels, and outputs the logit vector , where is the number of classes. The probability vector is then obtained as , where is the softmax function, and the predicted label is . Let denote the loss of the classifier with parameters on . When holding fixed and viewing the loss as a function of , we simply write .
2.2 Adversarial Examples
We consider a class of adversarial examples for image classifiers where small (imperceptible) perturbations are added to images to force the model to misclassify them. We consider the case where the norm of perturbation is bounded. The attacker’s problem is stated as follows:
(1)  
s.t. 
where and are the clean and adversarial samples, respectively, is the true label, and is the maximum allowed absolute change to each pixel.
Several optimizationbased methods have been proposed for generating adversarial examples, including fast gradient sign method [15], iterative gradient method [16], and Carlini and Wagner (CW) attack [5]. It has been observed that the specific choice of optimizer is less important than choosing to use iterative optimizationbased methods [4].
We generate adversarial examples using the Projected Gradient Descent (PGD) method [16, 4], which provides a unified framework for iterative attacks independent of the specific optimization function. Let be the number of steps and be the image at step . We have and . At step , the image is updated as
(2) 
where is the attack vector at step , is the added perturbation per step, and is the projection operator where is the set of allowed perturbations. In the case of bounded constraint, projector clips each pixel within of the corresponding pixel of original image .
The attack vector takes different forms depending on the attack goal. Generally, the attacker’s goal is to maximize the loss on the defender’s desired output or alternatively minimize the loss on attacker’s desired output. These approaches lead to two common attacks respectively known as misclassification and targeted attacks, for which attack vectors are specified as follows:

, for misclassification attack,

, for targeted attack,
where is the attacker’s desired target label.
3 Detecting Adversarial Examples Based on Logit Anomalies
In this section, we first review the logitbased statistical test proposed in [14] for detecting adversarial examples and then describe our classifierbased adaptation of the method.
3.1 Statistical Test Proposed in [14]
In [14], a method is proposed for detecting adversarial examples and estimating their true class. The method is based on a statistical test that measures how logits change under noise. Let , where denotes the logit of class for input . Consider , where is the predicted label and is the noise. Note that for , is negative and might increase with noise. The authors posited that common methods for generating adversarial examples result in perturbations that “overfit to specifics of and are not robust,” i.e., by adding noise to the adversarial example, the expected probability of the true class increases.
For an adversarial example and true label , if the adversarial perturbation is not robust to noise, we will have , meaning that adding noise will partially undo the effect of adversarial perturbation. This is the basis of the statistical test, i.e., we obtain the logits of noisy inputs and check whether the expected logit value of a label increased under noise. Such an input is detected to be adversarially perturbed and the label with increasing logit is output as the correct class. The details of the method is provided in the following.

Let . The normalized term is obtained as , where and are mean and standard deviation of computed on benign examples. Let .

Inputs are flagged as adversarially manipulated if
(3) where the thresholds are computed such that the true positive rate (TPR) is maximized, while the false positive rate (FPR) remains small (say ). If an input is detected to be an adversarial example, its correct label is predicted as .
3.2 Our ClassifierBased Detection Method
The detection method exploits the statistics of logits of clean and noisy inputs to distinguish between benign and adversarial samples. We note that instead of computing and the thresholds , we can design a generic binary classifier that takes logits of clean and noisy images as input and detects whether the image is adversarial. The advantage of using such a classifier is that it can identify anomalies in adversarial logits beyond their directionality toward the true label and, hence, improves the detection performance. Our classifierbased detection method is described in the following.

For validation data , generate adversarial samples .

Compute logits of clean inputs and the average logits of noisy inputs, i.e., compute , , , .

Construct vectors and , where denotes concatenation.

Train a detector classifier with inputoutput pairs and .
In experiments, we use a neural network with two hidden layers of size and ReLU activation for the detector. Note that the statistical test method can be exactly implemented by such a network as well. Therefore, we expect our approach to be able to better distinguish between benign and adversarial examples. Figure 2 illustrates our classifierbased detection method.
4 Bypassing LogitBased Detection Methods
In this section, we first review the attack approach of computing adversarial perturbations on noisy images and then propose the logit mimicry attack for bypassing the logitbased detection methods.
4.1 Attacking with Noisy Inputs
The statistical test method uses the property that adversarial examples are less robust to noise compared to benign examples. Based on this observation, [14] suggested that, to bypass the defense, the attacker needs to craft perturbations that perform well in expectation under noise. In this attack, the PGD update is computed based on the following loss function:
(4) 
where balances the attack updates for clean and noisy inputs. In experiments, we set and compute the loss as an empirical average over noisy inputs, with the same noise source used for detection.
4.2 Logit Mimicry Attack
Our classifierbased approach provides an additional insight into how the detection method can be bypassed. Recall that the detector takes the logits of clean images and the average logits of noisy images as input. Therefore, instead of computing perturbations that are robust to noise, the attacker can craft adversarial examples such that their logits mimic those of benign images. Specifically, an adversarial image that is classified as class must have similar logits to benign images that belong to the same class. Moreover, its expected logits of noisy examples must mimic average logits of noisy benign images that belong to that class. We call our attack Logit Mimicry Attack. The attack details are described in the following.

Let be a validation image and be the predicted label, i.e., . For each class , we compute the logit profiles of clean and noisy images respectively as follows:
(5) 
Given a test image , we aim to generate adversarial example with the target class . The attack losses on clean and noisy images are defined as follows:
(6) 
The logit mimicry attack loss is then obtained as
(7) where balances the loss on clean and noisy inputs. In experiments, we set .
Detection Method  CW  CW with Noisy Inputs  Logit Mimicry  

FPR  FPR  FPR  FPR  FPR  FPR  
Statistical Test  
Classifierbased  
5 Experimental Results
In this section, we present the experimental results of evaluating statistical test and classifierbased methods against different attacks.
5.1 Attack Setup
The experiments are performed on ResNet56 network [17] and CIFAR10 dataset [18]. The network achieves accuracy on CIFAR10 test data. As suggested in [14], for each image, we generate noisy images as , where and . Note that the pixel range is in . We train the detector with images of the CIFAR10 validation set, from which images are used for training and the rest are used for validation. We then test the detector on test images of CIFAR10 dataset.
The results are presented for targeted attack with PGD method. We generate adversarial examples using steps of PGD with perturbation of of the pixel dynamic range, as done in [14]. In all cases, we set the attack hyperparameters such that the attack success rate in evading the classifier is more than . Unless stated otherwise, we use the CW loss function [5] in our attacks, which maximizes the logit of the targeted class, while minimizing the largest value of other logits as
(8) 
where higher values of generates adversarial examples with higher confidence. We set in our experiments. We use CW’s approach, since it computes the loss using the logits and, hence, is more relevant to the detection method proposed in [14].
Threat Model. Various threat models have been considered for defending against evasion attacks. Commonly used threat models are whitebox (complete knowledge of the system) and blackbox (knowledge of training procedure and architecture of target model, but no access to the trained model itself). As stated in [19], “the guiding principle of a defense’s threat model is to assume that the adversary has complete knowledge of the inner workings of the defense.” That is, even in blackbox setting, it must be assumed that the attacker is fully aware of the defense mechanism. In the field of security, this guideline is derived from Kerckhoffs’s principle [20], which is stated also as “The enemy knows the system” by Claude Shannon [21].
In [14], the authors claimed robustness in whitebox setting. However, most of the analysis and experiments were performed for the case where the attacker is not aware of the defense. The authors only briefly discussed attacking with noisy inputs as a defenseaware attack, which we show is not the right countermeasure against the defense. In our experiments, we consider whitebox threat model and assume that the attacker is fully aware of the defense method.
5.2 Attack Results
CW Attack. Table 1 shows the attack results for different settings. As can be seen, when noise is large enough, the statistical test method achieves more than true positive rate (TPR) at and false positive rates (FPR). The classifierbased approach achieves higher TPR especially when the noise is too small, indicating that it can better identify anomalies in adversarial logits. The results verify the observation that adversarial examples that are generated by only maximizing the loss on the true class are less robust to noise compared to benign examples. Figure 3 visualizes this property by measuring the average prediction probabilities of different classes versus noise. It can be seen that, in CW adversarial examples, there is a range of , in which the true class is the most likely class, indicating that noisy CW adversarial examples are, indeed, biased toward the true class.
CW Attack with Noisy Inputs. As can be seen in Table 1, adding noise to inputs at each PGD step reduces the detection performance of the statistical test method, but the method still achieves high TPR. Also, the attack approach is less effective against the classifierbased method, suggesting that logits of adversarial examples generated by noisy CW attack contain anomalies that can be identified by a more complex detector.
Logit Mimicry Attack. The mimicry attack bypasses both statistical test and classifierbased methods, resulting in less than TPR in all cases. The results show that such statisticsbased defense mechanisms could be evaded by crafting adversarial examples that mimic the behavior of benign samples. Figure 3 shows the behavior of prediction probabilities versus noise and confirms that, unlike CW attack, in mimicry adversarial examples, the true class is never the most likely class, i.e., the directionality toward the true class has been eliminated. Figure 4 also shows samples of adversarial examples generated using CW and mimicry attacks. As can be seen, both attacks could evade the classifier by adding imperceptible perturbations to images.
6 Does There Exist Any LogitBased Robust Detector?
In previous section, we showed that a classifierbased detector that is trained with CW adversarial examples can be bypassed with logit mimicry attack. The detector can be similarly trained with adversarial examples obtained using mimicry attack. In this section, we investigate whether such a classifier can detect mimicry adversarial examples and how it can be attacked.
Training Detector with Mimicry Adversarial Examples. We follow the same approach of training a binary classifier proposed in Section 3.2, but generate adversarial examples using the logit mimicry attack. The attack results are provided in Table 2. As can be seen, the detector achieves TPR at FPR, indicating that the logits of mimicry adversarial examples are highly distinguishable from those of benign samples.
Adaptive Attacker. To evade the adaptive detector, the attacker must craft adversarial examples that fool both the main network and the detector. Specifically, adversarial examples must be generated by minimizing the network’s loss on targeted class and the detector’s loss on benign class. We use logit mimicry attack to evade the network, since it also helps in bypassing the detector. The attack method is described in the following.

Let be a function that takes a sample as input and outputs the logits. Let also be a binary classifier that takes logits of the clean sample and the average logits of noisy samples, and outputs zero if the image is benign and one otherwise. Let be the prediction probability of the benign class. We define the detector’s loss on sample as .

The overall attacker’s loss is then computed as
(9) where is defined in Equ. 7. In experiments, we set .
The adaptive attack achieves success rate in evading the network and, as can be seen in Table 2, reduces the TPR of the adaptive detector to .
Detection Method  Logit Mimicry Attack  Adaptive Attack  

FPR  FPR  FPR  FPR  
Adaptive Detector  
Iterativelytrained Detector 
IterativelyTrained Detector, Adaptive Attacker. We showed that adaptive attack successfully bypasses the adaptive detector. The question is does there exist any logitbased detector that is robust against adaptive attack? To answer this question, we iteratively update the detector against the adaptive attack, i.e., we train a detector and then iteratively run the adaptive attack against it and finetune the detector with the new adversarial examples. Once training is done, we run adaptive attack on final detector. In experiments, we train the detector for iterations. At each iteration, the detector is trained for five epochs with adaptive adversarial examples as well as benign samples. Figure 5 illustrates the training procedure of iterativelytrained detector.
As can be seen in Table 2, while such a detector achieves high TPR against the logit mimicry attack, it completely fails against the adaptive attacker. That is, even after iteratively training the classifier to detect adaptive adversarial examples, we can always find new examples that evade both network and detector. The results indicate that distinguishability of benign and adversarial logits is not a fundamental property of deep convolutional networks and, hence, cannot be used to detect adversarial examples.
7 Conclusion
A recent paper presented at ICML 2019 proposed a statistical test method for detecting adversarial examples. The method is designed based on the observation that logits of noisy adversarial examples “tend to have a characteristic direction” toward the true class, whereas it does not have any specific direction if the input is benign. In this paper, we first developed a classifierbased adaptation of the statistical test method, which improves the detection performance. We then proposed the Logit Mimicry Attack (LMA) method to generate adversarial examples such that the logits of clean and noisy images mimic the distribution of those of benign samples. We showed that LMA successfully bypasses both the statistical test and classifierbased methods. We finally evaluated the robustness of a detector that is iteratively trained to detect adversarial examples that are specifically generated to bypass it. We showed that even such a detector fails against the adaptive attack, indicating that adversarial logits can be made to mimic any characteristic behavior of benign logits.
References
 [1] I. Goodfellow, N. Papernot, S. Huang, Y. Duan, and P. Abbeel, “Attacking machine learning with adversarial examples,” Open AI Blog, 2017. https://blog.openai.com/adversarialexampleresearch/.
 [2] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” in Joint European conference on machine learning and knowledge discovery in databases, pp. 387–402, Springer, 2013.
 [3] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
 [4] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
 [5] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57, IEEE, 2017.
 [6] J. Uesato, B. OâDonoghue, P. Kohli, and A. Oord, “Adversarial risk and the dangers of evaluating against weak attacks,” in International Conference on Machine Learning, pp. 5032–5041, 2018.
 [7] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in International Conference on Machine Learning, pp. 274–283, 2018.
 [8] A. Athalye and N. Carlini, “On the robustness of the cvpr 2018 whitebox adversarial example defenses,” arXiv preprint arXiv:1804.03286, 2018.
 [9] L. Engstrom, A. Ilyas, and A. Athalye, “Evaluating and understanding the robustness of adversarial logit pairing,” arXiv preprint arXiv:1807.10272, 2018.
 [10] N. Carlini, “Is AmI (attacks meet interpretability) robust to adversarial examples?,” arXiv preprint arXiv:1902.02322, 2019.
 [11] W. He, J. Wei, X. Chen, N. Carlini, and D. Song, “Adversarial example defense: Ensembles of weak defenses are not strong,” in 11th USENIX Workshop on Offensive Technologies (WOOT 17), 2017.
 [12] N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3–14, ACM, 2017.
 [13] N. Carlini and D. Wagner, “Magnet and ”efficient defenses against adversarial attacks” are not robust to adversarial examples,” arXiv preprint arXiv:1711.08478, 2017.
 [14] K. Roth, Y. Kilcher, and T. Hofmann, “The odds are odd: A statistical test for detecting adversarial examples,” in International Conference on Machine Learning, pp. 5498–5507, 2019.
 [15] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
 [16] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” arXiv preprint arXiv:1611.01236, 2016.
 [17] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European conference on computer vision, pp. 630–645, Springer, 2016.
 [18] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009.
 [19] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, and A. Madry, “On evaluating adversarial robustness,” arXiv preprint arXiv:1902.06705, 2019.
 [20] D. Kahn, The Codebreakers: The comprehensive history of secret communication from ancient times to the internet. Simon and Schuster, 1996.
 [21] C. E. Shannon, “Communication theory of secrecy systems,” Bell system technical journal, vol. 28, no. 4, pp. 656–715, 1949.