Abstract
We investigate a family of poisoning attacks against Support Vector Machines (SVM). Such attacks inject specially crafted training data that increases the SVM’s test error. Central to the motivation for these attacks is the fact that most learning algorithms assume that their training data comes from a natural or wellbehaved distribution. However, this assumption does not generally hold in securitysensitive settings. As we demonstrate, an intelligent adversary can, to some extent, predict the change of the SVM’s decision function due to malicious input and use this ability to construct malicious data.
The proposed attack uses a gradient ascent strategy in which the gradient is computed based on properties of the SVM’s optimal solution. This method can be kernelized and enables the attack to be constructed in the input space even for nonlinear kernels. We experimentally demonstrate that our gradient ascent procedure reliably identifies good local maxima of the nonconvex validation error surface, which significantly increases the classifier’s test error.
Poisoning Attacks against Support Vector Machines
Battista Biggio battista.biggio@diee.unica.it
Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d’Armi, 09123 Cagliari, Italy
Blaine Nelson blaine.nelson@wsii.unituebingen.de
Pavel Laskov pavel.laskov@unituebingen.de
Wilhelm Schickard Institute for Computer Science, University of Tübingen, Sand 1, 72076 Tübingen, Germany
Machine learning techniques are rapidly emerging as a vital tool in a variety of networking and largescale system applications because they can infer hidden patterns in large complicated datasets, adapt to new behaviors, and provide statistical soundness to decisionmaking processes. Application developers thus can employ learning to help solve socalled bigdata problems and these include a number of securityrelated problems particularly focusing on identifying malicious or irregular behavior. In fact, learning approaches have already been used or proposed as solutions to a number of such securitysensitive tasks including spam, worm, intrusion and fraud detection (Meyer & Whateley, 2004; Biggio et al., 2010; Stolfo et al., 2003; Forrest et al., 1996; Bolton & Hand, 2002; Cova et al., 2010; Rieck et al., 2010; Curtsinger et al., 2011; Laskov & Šrndić, 2011). Unfortunately, in these domains, data is generally not only nonstationary but may also have an adversarial component, and the flexibility afforded by learning techniques can be exploited by an adversary to achieve his goals. For instance, in spamdetection, adversaries regularly adapt their approaches based on the popular spam detectors, and generally a clever adversary will change his behavior either to evade or mislead learning.
In response to the threat of adversarial data manipulation, several proposed learning methods explicitly account for certain types of corrupted data (Globerson & Roweis, 2006; Teo et al., 2008; Brückner & Scheffer, 2009; Dekel et al., 2010). Attacks against learning algorithms can be classified, among other categories (c.f. Barreno et al., 2010), into causative (manipulation of training data) and exploratory (exploitation of the classifier). Poisoning refers to a causative attack in which specially crafted attack points are injected into the training data. This attack is especially important from the practical point of view, as an attacker usually cannot directly access an existing training database but may provide new training data; e.g., webbased repositories and honeypots often collect malware examples for training, which provides an opportunity for the adversary to poison the training data. Poisoning attacks have been previously studied only for simple anomaly detection methods (Barreno et al., 2006; Rubinstein et al., 2009; Kloft & Laskov, 2010).
In this paper, we examine a family of poisoning attacks against Support Vector Machines (SVM). Following the general security analysis methodology for machine learning, we assume that the attacker knows the learning algorithm and can draw data from the underlying data distribution. Further, we assume that our attacker knows the training data used by the learner; generally, an unrealistic assumption, but in realworld settings, an attacker could instead use a surrogate training set drawn from the same distribution (e.g., Nelson et al., 2008) and our approach yields a worstcase analysis of the attacker’s capabilities. Under these assumptions, we present a method that an attacker can use to construct a data point that significantly decreases the SVM’s classification accuracy.
The proposed method is based on the properties of the optimal solution of the SVM training problem. As was first shown in an incremental learning technique (Cauwenberghs & Poggio, 2001), this solution depends smoothly on the parameters of the respective quadratic programming problem and on the geometry of the data points. Hence, an attacker can manipulate the optimal SVM solution by inserting specially crafted attack points. We demonstrate that finding such an attack point can be formulated as optimization with respect to a performance measure, subject to the condition that an optimal solution of the SVM training problem is retained. Although the test error surface is generally nonconvex, the gradient ascent procedure used in our method reliably identifies good local maxima of the test error surface.
The proposed method only depends on the gradients of the dot products between points in the input space, and hence can be kernelized. This contrasts previous work involving construction of special attack points (e.g., Brückner & Scheffer, 2009; Kloft & Laskov, 2010) in which attacks could only be constructed in the feature space for the nonlinear case. The latter is a strong disadvantage for the attacker, since he must construct data in the input space and has no practical means to access the feature space. Hence, the proposed method breaks new ground in optimizing the impact of datadriven attacks against kernelbased learning algorithms and emphasizes the need to consider resistance against adversarial training data as an important factor in the design of learning algorithms.
We assume the SVM has been trained on a data set , . Following the standard notation, denotes the matrix of kernel values between two sets of points, denotes the labelannotated version of , and denotes the SVM’s dual variables corresponding to each training point. Depending on the value of , the training points are referred to as margin support vectors (, set ), error support vectors (, set ) and reserve points (, set ). In the sequel, the lowercase letters are used to index the corresponding parts of vectors or matrices; e.g., denotes the margin support vector submatrix of .
For a poisoning attack, the attacker’s goal is to find a point , whose addition to maximally decreases the SVM’s classification accuracy. The choice of the attack point’s label, , is arbitrary but fixed. We refer to the class of this chosen label as attacking class and the other as the attacked class.
The attacker proceeds by drawing a validation data set and maximizing the hinge loss incurred on by the SVM trained on :
(1) 
In this section, we assume the role of the attacker and develop a method for optimizing with this objective.
First, we explicitly account for all terms in the margin conditions that are affected by :
(2)  
It is not difficult to see from the above equations that is a nonconvex objective function. Thus, we exploit a gradient ascent technique to iteratively optimize it. We assume that an initial location of the attack point has been chosen. Our goal is to update the attack point as where is the current iteration, is a norm1 vector representing the attack direction, and is the step size. Clearly, to maximize our objective, the attack direction aligns to the gradient of with respect to , which has to be computed at each iteration.
Although the hinge loss is not everywhere differentiable, this can be overcome by only considering point indices with nonzero contributions to ; i.e., those for which . Contributions of such points to the gradient of can be computed by differentiating Eq. (2) with respect to using the product rule:
(3) 
where
The expressions for the gradient can be further refined using the fact that the step taken in direction should maintain the optimal SVM solution. This can expressed as an adiabatic update condition using the technique introduced in (Cauwenberghs & Poggio, 2001). Observe that for the th point in the training set, the KKT conditions for the optimal solution of the SVM training problem can be expressed as:
(4)  
(5) 
The equality in condition (4) and (5) implies that an infinitesimal change in the attack point causes a smooth change in the optimal solution of the SVM, under the restriction that the composition of the sets , and remain intact. This equilibrium allows us to predict the response of the SVM solution to the variation of , as shown below.
By differentiation of the dependent terms in Eqs. (4)–(5) with respect to each component (), we obtain, for any ,
(6)  
which can be rewritten as
(7) 
The first matrix can be inverted using the ShermanMorrisonWoodbury formula (Lütkepohl, 1996):
(8) 
where and . Substituting (8) into (7) and observing that all components of the inverted matrix are independent of , we obtain:
(9)  
Substituting (9) into (3) and further into (1), we obtain the desired gradient used for optimizing our attack:
(10) 
where
From Eq. (10), we see that the gradient of the objective function at iteration may depend on the attack point only through the gradients of the matrix . In particular, this depends on the chosen kernel. We report below the expressions of these gradients for three common kernels.

Linear kernel:

Polynomial kernel:

RBF kernel:
The dependence on (and, thus, on ) in the gradients of nonlinear kernels can be avoided by substituting with , provided that is sufficiently small. This approximation enables a straightforward extension of our method to arbitrary kernels.
The algorithmic details of the method described in Section id1 are presented in Algorithm 1.
In this algorithm, the attack vector is initialized by cloning an arbitrary point from the attacked class and flipping its label. In principle, any point sufficiently deep within the attacking class’s margin can be used as a starting point. However, if this point is too close to the boundary of the attacking class, the iteratively adjusted attack point may become a reserve point, which halts further progress.
The computation of the gradient of the validation error crucially depends on the assumption that the structure of the sets , and does not change during the update. In general, it is difficult to determine the largest step along an arbitrary direction , which preserves this structure. The classical line search strategy used in gradient ascent methods is not suitable for our case, since the update to the optimal solution for large steps may be prohibitively expensive. Hence, the step is fixed to a small constant value in our algorithm. After each update of the attack point , the optimal solution is efficiently recomputed from the solution on , using the incremental SVM machinery (e.g., Cauwenberghs & Poggio, 2001).
The algorithm terminates when the change in the validation error is smaller than a predefined threshold. For kernels including the linear kernel, the surface of the validation error is unbounded, hence the algorithm is halted when the attack vector deviates too much from the training data; i.e., we bound the size of our attack points.
The experimental evaluation presented in the following sections demonstrates the behavior of our proposed method on an artificial twodimensional dataset and evaluates its effectiveness on the classical MNIST handwritten digit recognition dataset.
We first consider a twodimensional data generation model in which each class follows a Gaussian distribution with mean and covariance matrices given by , , . The points from the negative distribution are assigned the label (shown as red in the subsequent figures) and otherwise (shown as blue). The training and the validation sets, and (consisting of and points per class, respectively) are randomly drawn from this distribution.
In the experiment presented below, the red class is the attacking class. To this end, a random point of the blue class is selected and its label is flipped to serve as the starting point for our method. Our gradient ascent method is then used to refine this attack until its termination condition is satisfied. The attack’s trajectory is traced as the black line in Fig. 1 for both the linear kernel (upper two plots) and the RBF kernel (lower two plots). The background in each plot represents the error surface explicitly computed for all points within the box . The leftmost plots in each pair show the hinge loss computed on a validation set while the rightmost plots in each pair show the classification error for the area of interest. For the linear kernel, the range of attack points is limited to the box shown as a dashed line.
For both kernels, these plots show that our gradient ascent algorithm finds a reasonably good local maximum of the nonconvex error surface. For the linear kernel, it terminates at the corner of the bounded region, since the error surface is unbounded. For the RBF kernel, it also finds a good local maximum of the hinge loss which, incidentally, is the maximum classification error within this area of interest.
We now quantitatively validate the effectiveness of the proposed attack strategy on a wellknown MNIST handwritten digit classification task (LeCun et al., 1995). Similarly to Globerson & Roweis (2006), we focus on twoclass subproblems of discriminating between two distinct digits.^{1}^{1}1The data set is also publicly available in Matlab format at http://cs.nyu.edu/~roweis/data.html. In particular, we consider the following twoclass problems: 7 vs. 1; 9 vs. 8; 4 vs. 0. The visual nature of the handwritten digit data provides us with a semantic meaning for an attack.
Each digit in the MNIST data set is properly normalized and represented as a grayscale image of pixels. In particular, each pixel is ordered in a rasterscan and its value is directly considered as a feature. The overall number of features is . We normalized each feature (pixel value) by dividing its value by .
In this experiment only the linear kernel is considered, and the regularization parameter of the SVM is fixed to . We randomly sample a training and a validation data of and samples, respectively, and retain the complete testing data given by MNIST for . Although it varies for each digit, the size of the testing data is about 2000 samples per class (digit).
The results of the experiment are presented in Fig. 2. The leftmost plots of each row show the example of the attacked class taken as starting points in our algorithm. The middle plots show the final attack point. The rightmost plots displays the increase in the validation and testing errors as the attack progresses.
The visual appearance of the attack point reveals that the attack blurs the initial prototype toward the appearance of examples of the attacking class. Comparing the initial and final attack points, we see this effect: the bottom segment of the straightens to resemble a , the lower segment of the becomes more round thus mimicking an , and round noise is added to the outer boundary of the to make it similar to a .
The increase in error over the course of attack is especially striking, as shown in the rightmost plots. In general, the validation error overestimates the classification error due to a smaller sample size. Nonetheless, in the exemplary runs reported in this experiment, a single attack data point caused the classification error to rise from the initial error rates of 2–5% to 15–20%. Since our initial attack point is obtained by flipping the label of a point in the attacked class, the errors in the first iteration of the rightmost plots of Fig. 2 are caused by single random label flips. This confirms that our attack can achieve significantly higher error rates than random label flips, and underscores the vulnerability of the SVM to poisoning attacks.
The latter point is further illustrated in a multiple point, multiple run experiment presented in Fig. 3. For this experiment, the attack was extended by injecting additional points into the same class and averaging results over multiple runs on randomly chosen training and validation sets of the same size (100 and 500 samples, respectively). One can clearly see a steady growth of the attack effectiveness with the increasing percentage of the attack points in the training set. The variance of the error is quite high, which can be explained by relatively small sizes of the training and validation data sets.
The poisoning attack presented in this paper is the first step toward the security analysis of SVM against training data attacks. Although our gradient ascent method is arguably a crude algorithmic procedure, it attains a surprisingly large impact on the SVM’s empirical classification accuracy. The presented attack method also reveals the possibility for assessing the impact of transformations carried out in the input space on the functions defined in the Reproducing Kernel Hilbert Spaces by means of differential operators. Compared to previous work on evasion of learning algorithms (e.g., Brückner & Scheffer, 2009; Kloft & Laskov, 2010), such influence may facilitate the practical realization of various evasion strategies. These implications need to be further investigated.
Several potential improvements to the presented method remain to be explored in future work. The first would be to address our optimization method’s restriction to small changes in order to maintain the SVM’s structural constraints. We solved this by taking many tiny gradient steps. It would be interesting to investigate a more accurate and efficient computation of the largest possible step that does not alter the structure of the optimal solution.
Another direction for research is the simultaneous optimization of multipoint attacks, which we successfully approached with sequential singlepoint attacks. The first question is how to optimally perturb a subset of the training data; that is, instead of individually optimizing each attack point, one could derive simultaneous steps for every attack point to better optimize their overall effect. The second question is how to choose the best subset of points to use as a starting point for the attack. Generally, the latter is a subset selection problem but heuristics may allow for improved approximations. Regardless, we demonstrate that even nonoptimal multipoint attack strategies significantly degrade the SVM’s performance.
An important practical limitation of the proposed method is the assumption that the attacker controls the labels of the injected points. Such assumptions may not hold when the labels are only assigned by trusted sources such as humans. For instance, a spam filter uses its users’ labeling of messages as its ground truth. Thus, although an attacker can send arbitrary messages, he cannot guarantee that they will have the labels necessary for his attack. This imposes an additional requirement that the attack data must satisfy certain side constraints to fool the labeling oracle. Further work is needed to understand these potential side constraints and to incorporate them into attacks.
The final extension would be to incorporate the realworld inverse featuremapping problem; that is, the problem of finding realworld attack data that can achieve the desired result in the learner’s input space. For data like handwritten digits, there is a direct mapping between the realworld image data and the input features used for learning. In many other problems (e.g., spam filtering) the mapping is more complex and may involve various nonsmooth operations and normalizations. Solving these inverse mapping problems for attacks against learning remains open.
Acknowledgments
This work was supported by a grant awarded to B. Biggio by Regione Autonoma della Sardegna, and by the project No. CRP18293 funded by the same institution, PO Sardegna FSE 20072013, L.R. 7/2007 “Promotion of the scientific research and technological innovation in Sardinia”. The authors also wish to acknowledge the Alexander von Humboldt Foundation and the Heisenberg Fellowship of the Deutsche Forschungsgemeinschaft (DFG) for providing financial support to carry out this research. The opinions expressed in this paper are solely those of the authors and do not necessarily reflect the opinions of any sponsor.
References
 Barreno et al. (2006) Barreno, Marco, Nelson, Blaine, Sears, Russell, Joseph, Anthony D., and Tygar, J. D. Can machine learning be secure? In Proceedings of the ACM Symposium on Information, Computer and Communications Security (ASIACCS), pp. 16–25, 2006.
 Barreno et al. (2010) Barreno, Marco, Nelson, Blaine, Joseph, Anthony D., and Tygar, J. D. The security of machine learning. Machine Learning, 81(2):121–148, November 2010.
 Biggio et al. (2010) Biggio, Battista, Fumera, Giorgio, and Roli, Fabio. Multiple classifier systems for robust classifier design in adversarial environments. International Journal of Machine Learning and Cybernetics, 1(1):27–41, 2010.
 Bolton & Hand (2002) Bolton, Richard J. and Hand, David J. Statistical fraud detection: A review. Journal of Statistical Science, 17(3):235–255, 2002.
 Brückner & Scheffer (2009) Brückner, Michael and Scheffer, Tobias. Nash equilibria of static prediction games. In Advances in Neural Information Processing Systems (NIPS), pp. 171–179. 2009.
 Cauwenberghs & Poggio (2001) Cauwenberghs, Gert and Poggio, Tomaso. Incremental and decremental support vector machine learning. In Leen, T.K., Diettrich, T.G., and Tresp, V. (eds.), Advances in Neural Information Processing Systems 13, pp. 409–415, 2001.
 Cova et al. (2010) Cova, M., Kruegel, C., and Vigna, G. Detection and analysis of drivebydownload attacks and malicious JavaScript code. In International Conference on World Wide Web (WWW), pp. 281–290, 2010.
 Curtsinger et al. (2011) Curtsinger, C., Livshits, B., Zorn, B., and Seifert, C. ZOZZLE: Fast and precise inbrowser JavaScript malware detection. In USENIX Security Symposium, pp. 33–48, 2011.
 Dekel et al. (2010) Dekel, O., Shamir, O., and Xiao, L. Learning to classify with missing and corrupted features. Machine Learning, 81(2):149–178, 2010.
 Forrest et al. (1996) Forrest, Stephanie, Hofmeyr, Steven A., Somayaji, Anil, and Longstaff, Thomas A. A sense of self for unix processes. In Proceedings of the IEEE Symposium on Security and Privacy, pp. 120–128, 1996.
 Globerson & Roweis (2006) Globerson, A. and Roweis, S. Nightmare at test time: Robust learning by feature deletion. In International Conference on Machine Learning (ICML), pp. 353–360, 2006.
 Kloft & Laskov (2010) Kloft, Marius and Laskov, Pavel. Online anomaly detection under adversarial impact. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
 Laskov & Šrndić (2011) Laskov, Pavel and Šrndić, Nedim. Static detection of malicious JavaScriptbearing PDF documents. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), December 2011.
 LeCun et al. (1995) LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Müller, U., Säckinger, E., Simard, P., and Vapnik, V. Comparison of learning algorithms for handwritten digit recognition. In Int’l Conf. on Artificial Neural Networks, pp. 53–60, 1995.
 Lütkepohl (1996) Lütkepohl, Helmut. Handbook of matrices. John Wiley & Sons, 1996.
 Meyer & Whateley (2004) Meyer, Tony A. and Whateley, Brendon. SpamBayes: Effective opensource, Bayesian based, email classification system. In Proceedings of the Conference on Email and AntiSpam (CEAS), July 2004.
 Nelson et al. (2008) Nelson, Blaine, Barreno, Marco, Chi, Fuching Jack, Joseph, Anthony D., Rubinstein, Benjamin I. P., Saini, Udam, Sutton, Charles, Tygar, J. D., and Xia, Kai. Exploiting machine learning to subvert your spam filter. In Proceedings of the 1st USENIX Workshop on LargeScale Exploits and Emergent Threats (LEET), pp. 1–9, 2008.
 Rieck et al. (2010) Rieck, K., Krüger, T., and Dewald, A. Cujo: Efficient detection and prevention of drivebydownload attacks. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), pp. 31–39, 2010.
 Rubinstein et al. (2009) Rubinstein, Benjamin I. P., Nelson, Blaine, Huang, Ling, Joseph, Anthony D., hon Lau, Shing, Rao, Satish, Taft, Nina, and Tygar, J. D. ANTIDOTE: Understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement (IMC), pp. 1–14, 2009.
 Stolfo et al. (2003) Stolfo, Salvatore J., Hershkop, Shlomo, Wang, Ke, Nimeskern, Olivier, and Hu, ChiaWei. A behaviorbased approach to securing email systems. In Mathematical Methods, Models and Architectures for Computer Networks Security. SpringerVerlag, 2003.
 Teo et al. (2008) Teo, C.H., Globerson, A., Roweis, S., and Smola, A. Convex learning with invariances. In Advances in Neural Information Proccessing Systems (NIPS), pp. 1489–1496, 2008.