We study the problem of learning with label proportions in which the training data is provided in groups and only the proportion of each class in each group is known. We propose a new method called proportion-SVM, or SVM, which explicitly models the latent unknown instance labels together with the known group label proportions in a large-margin framework. Unlike the existing works, our approach avoids making restrictive assumptions about the data. The SVM model leads to a non-convex integer programming problem. In order to solve it efficiently, we propose two algorithms: one based on simple alternating optimization and the other based on a convex relaxation. Extensive experiments on standard datasets show that SVM outperforms the state-of-the-art, especially for larger group sizes.
SVM for Learning with Label Proportions
Felix X. Yu email@example.com
Dong Liu firstname.lastname@example.org
Sanjiv Kumar email@example.com
Tony Jebara firstname.lastname@example.org
Shih-Fu Chang email@example.com
Columbia University, New York, NY 10027
Google Research, New York, NY 10011
The problem of learning with label proportions has recently drawn attention in the learning community (Quadrianto et al., 2009; Rüeping, 2010). In this setting, the training instances are provided as groups or “bags”. For each bag, only the proportions of the labels are available. The task is to learn a model to predict labels of the individual instances.
Learning with label proportions raises multiple issues. On one hand, it enables interesting applications such as modeling voting behaviors from aggregated proportions across different demographic regions. On the other hand, the feasibility of such a learning method also raises concerns about the sensitive personal information that could potentially be leaked simply by observing label proportions.
To address this learning setting, this article explicitly models the unknown instance labels as latent variables. This alleviates the need for making restrictive assumptions on the data, either parametric or generative. We introduce a large-margin framework called proportion-SVM, or SVM111 is the symbol for “proportional-to”., which jointly optimizes over the unknown instance labels and the known label proportions (Section id1). In order to solve SVM efficiently, we propose two algorithms - one based on simple alternating optimization (Section id1), and the other based on a convex relaxation (Section id1). We show that our approach outperforms the existing methods for various datasets and settings (Section id1). The gains are especially higher for more challenging settings when the bag size is large.
MeanMap: Quadrianto et al. (2009) proposed a theoretically sound method to estimate the mean of each class using the mean of each bag and the label proportions. These estimates are then used in a conditional exponential model to maximize the log likelihood. The key assumption in MeanMap is that the class-conditional distribution of data is independent of the bags. Unfortunately, this assumption does not hold for many real world applications. For example, in modeling voting behaviors, in which the bags are different demographic regions, the data distribution can be highly dependent on the bags.
Inverse Calibration (InvCal): Rüeping (2010) proposed treating the mean of each bag as a “super-instance”, which was assumed to have a soft label corresponding to the label proportion. The “super-instances” can be poor in representing the properties of the bags. Our work also utilizes a large-margin framework, but we explicitly model the instance labels. Section id1 gives a detailed comparison with InvCal.
Figure 1 provides a toy example to highlight the problems with MeanMap and InvCal, which are the state-of-the-art methods.
Related Learning Settings: In semi-supervised learning, Mann and McCallum (2007) and Bellare et al. (2009) used an expectation regularization term to encourage model predictions on the unlabeled data to match the given proportions. Similar ideas were also studied in the generalized regularization method (Gillenwater et al., 2011). Li et al. (2009a) proposed a variant of semi-supervised SVM to incorporate the label mean of the unlabeled data. Unlike semi-supervised learning, the learning setting we are considering requires no instance labels for training.
As an extension to multiple-instance learning, Kuck and de Freitas (2005) designed a hierarchical probabilistic model to generate consistent label proportions. Besides the inefficiency in optimization, the method was shown to be inferior to MeanMap (Quadrianto et al., 2009). Similar ideas have also been studied by Chen et al. (2006) and Musicant et al. (2007).
Stolpe and Morik (2011) proposed an evolutionary strategy paired with a labeling heuristic for clustering with label proportions. Different from clustering, the proposed SVM framework jointly optimizes the latent instance labels and a large-margin classification model. The SVM formulation is related to large-margin clustering (Xu et al., 2004), with an additional objective to utilize the label proportions. Specifically, the convex relaxation method we used is inspired by the works of Li et al. (2009a) and Xu et al. (2004).
We consider a binary learning setting as follows. The training set is given in the form of bags,
In this paper, we assume that the bags are disjoint, i.e., , . The -th bag is with label proportion :
in which denotes the unknown ground-truth label of , . We use for predicting the binary label of an instance , where is a map of the input data.
We explicitly model the unknown instance labels as , in which denotes the unknown label of , . Thus the label proportion of the -th bag can be straightforwardly modeled as
We formulate the SVM under the large-margin framework as below.
in which is the loss function for classic supervised learning. is a function to penalize the difference between the true label proportion and the estimated label proportion based on . The task is to simultaneously optimize the labels and the model parameters and .
The above formulation permits using different loss functions for and . One can also add weights for different bags. Throughout this paper, we consider as the hinge loss, which is widely used for large-margin learning: . The algorithms in Section id1 and Section id1 can be easily generalized to different .
Compared to (Rüeping, 2010; Quadrianto et al., 2009), SVM requires no restrictive assumptions on the data. In fact, in the special case where no label proportions are provided, SVM becomes large-margin clustering (Xu et al., 2004; Li et al., 2009a), whose solution depends only on the data distribution. SVM can naturally incorporate any amount of supervised data without modification. The labels for such instances will be observed variables instead of being hidden. SVM can be easily extended to the multi-class case, similar to (Keerthi et al., 2012).
As stated in Section id1, the Inverse Calibration method (InvCal) (Rüeping 2010) treats the mean of each bag as a “super-instance”, which is assumed to have a soft label corresponding to the label proportion. It is formulated as below.
in which the -th bag mean is , . Unlike SVM, the proportion of the -th bag is modeled on top of this “super-instance” as:
The second term of the objective function (5) tries to impose , , albeit in an inverse way.
Though InvCal is shown to outperform other alternatives, including MeanMap (Quadrianto et al., 2009) and several simple large-margin heuristics, it has a crucial limitation. Note that (6) is not a good way of measuring the proportion predicted by the model, especially when the data has high variance, or the data distribution is dependent on the bags. In our formulation (S3.Ex1), by explicitly modeling the unknown instance labels , the label proportion can be directly modeled as given in (3). The advantage of our method is illustrated in a toy experiment shown in Figure 1 (for details see Section id1).
The SVM formulation is fairly intuitive and straightforward. It is, however, a non-convex integer programming problem, which is NP-hard. Therefore, one key issue lies in how to find an efficient algorithm to solve it approximately. In this paper, we provide two solutions: a simple alternating optimization method (Section id1), and a convex relaxation method (Section id1).
In SVM, the unknown instance labels can be seen as a bridge between supervised learning loss and label proportion loss. Therefore, one natural way for solving (S3.Ex1) is via alternating optimization as,
For a fixed , the optimization of (S3.Ex1) and becomes a classic SVM problem.
When and are fixed, the problem becomes:
We show that the second step above can be solved efficiently. Because the influence of each bag , on the objective is independent, we can optimize (S4.Ex6) on each bag separately. In particular, solving yields the following optimization problem:
For a fixed , (S4.Ex7) can be optimally solved by the steps below.
Initialize , . The optimal solution can be obtained by flipping the signs as below.
By flipping the sign of , , suppose the reduction of the first term in (S4.Ex7) is . Sort , . Then flip the signs of the top- ’s which have the highest reduction . .
For bag , we only need to sort the corresponding , once. Sorting takes time. After that, for each , the optimal solution can be computed incrementally, each taking time. We then pick the solution with the smallest objective value, yielding the optimal solution of (S4.Ex7).
Following the above steps, (S4.Ex6) can be solved in time, .
The proofs of the above propositions are given in the supplementary material.
By alternating between solving (, ) and , the objective is guaranteed to converge. This is due to the fact that the objective function is lower bounded, and non-increasing. In practice, we terminate the procedure when the objective no longer decreases (or if its decrease is smaller than a threshold). Empirically, the alternating optimization typically terminates fast within tens of iterations, but one obvious problem is the possibility of local solutions.
To alleviate this problem, similar to T-SVM (Joachims, 1999; Chapelle et al., 2008), the proposed alter-SVM algorithm (Algorithm 1) takes an additional annealing loop to gradually increase . Because the nonconvexity of the objective function mainly comes from the second term of (S3.Ex1), the annealing can be seen as a “smoothing” step to protect the algorithm from sub-optimal solutions. Following (Chapelle et al., 2008), we set in Algorithm 1 throughout this work. The convergence and annealing are further discussed in the supplementary material.
In the implementation of alter-SVM, we consider as the absolute loss: . Empirically, each alter-SVM loop given an annealing value terminates within a few iterations. From Proposition 2, optimizing has linear complexity in (when is small). Therefore the overall complexity of the algorithm depends on the SVM solver. Specifically, when linear SVM is used (Joachims, 2006), alter-SVM has linear complexity. In practice, to further alleviate the influence of the local solutions, similar to clustering, e.g., kmeans, we repeat alter-SVM multiple times by randomly initializing , and then picking the solution with the smallest objective value.
In this section, we show that with proper relaxation of the SVM formulation (S3.Ex1), the objective function can be transformed to a convex function of . We then relax the solution space of to its convex hull, leading to a convex optimization problem of . The conv-SVM algortihm is proposed to solve the relaxed problem. Unlike alter-SVM, conv-SVM does not require multiple initializations. This method is motivated by the techniques used in large-margin clustering (Li et al., 2009b; Xu et al., 2004).
We change the label proportion term in the objective function (S3.Ex1) as a constraint , and we drop the bias term 222If the bias term is not dropped, there will be constraint in the dual, leading to non-convexity. Such difficulty has also been discussed in (Xu et al., 2004). Fortunately, the effect of removing the bias term can be alleviated by zero-centering the data or augmenting the feature vector with an additional dimension with value .. Then, (S3.Ex1) is rewritten as:
in which controls the compatibility of the label proportions. The constraint can be seen as a special loss function:
We then write the inner problem of (9) as its dual:
in which , denotes pointwise-multiplication, is the kernel matrix with , , and .
which is a non-convex set. In order to get a convex optimization problem, we relax to its convex hull, the tightest convex relaxation of :
in which .
Thus solving the relaxed is identical to finding :
Fortunately, we can assume that at optimality only a small number of ’s are active in (14). Define as the set containing all the active ’s. We show that can be incrementally found by the cutting plane method.
It is easy to verify that the above is equivalent to:
This form enables us to apply the cutting plane method (Kelley Jr, 1960) to incrementally include the most violated into , and then solve the MKL problem, (14) with replaced as . The above can be repeated until no violated exists.
In the cutting plane training, the critical step is to obtain the most violated :
which is equivalent to
This is a 0/1 concave QP, for which there exists no efficient solution. However, instead of finding the most violated constraint, if we find any violated constraint , the objective function still decreases. We therefore relax the objective in (18), which can be solved efficiently. Note that the objective of (18) is equivalent to a norm . Following (Li et al., 2009b), we approximate it as the norm:
in which is the -th dimension of the -th feature vector. These can be obtained by eigendecomposition of the kernel matrix , when a nonlinear kernel is used. The computational complexity is . In practice, we choose such that 90% of the variance is preserved. We further rewrite (19) as:
Therefore the approximation from (18) to (19) enables us to consider each dimension and each bag separately. For the -th dimension, and the -th bag, we only need to solve two sub-problems , and . The former, as an example, can be written as
(18) with the norm approximated as the norm can be solved in time, .
As an additional step, we need to recover from . This is achieved by rank-1 approximation of (as )333Note that . This ambiguity can be resolved by validation on the training bags.. Because of the convex relaxation, the computed is not binary. However, we can use the real-valued directly in our prediction model (with dual):
Similar to alter-SVM, the objective of conv-SVM is guaranteed to converge. In practice, we terminate the algorithm when the decrease of the objective is smaller than a threshold. Typically the SimpleMKL converges in less than 5 iterations, and conv-SVM terminates in less than 10 iterations. The SimpleMKL takes (computing the gradient) time, or the complexity of SVM, whichever is higher. Recovering takes time and computing eigendecomposition with the first singular values takes time.
MeanMap (Quadrianto et al., 2009) was shown to outperform alternatives including kernel density estimation, discriminative sorting and MCMC (Kuck and de Freitas, 2005). InvCal (Rüeping, 2010) was shown to outperform MeanMap and several large-margin alternatives. Therefore, in the experiments, we only compare our approach with MeanMap and InvCal.
To visually demonstrate the advantage of our approach, we first show an experiment on a toy dataset with two bags. Figure 1 (a) and (b) show the data of the two bags, and Figure 1 (c) and (d) show the learned separating hyperplanes from different methods. Linear kernel is used in this experiment. For this specific dataset, the restrictive data assumptions of MeanMap and InvCal do not hold: the mean of the first bag (60% positive) is on the “negative side”, whereas, the mean of the second bag (40% positive) is on the “positive side”. Consequently, both MeanMap and InvCal completely fail, with the classification accuracy of 0%. On the other hand, our method, which does not make strong data assumptions, achieves the perfect performance with 100% accuracy.
Datasets. We compare the performance of different techniques on various datasets from the UCI repository444http://archive.ics.uci.edu/ml/ and the LibSVM collection555http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/. The details of the datasets are listed in Table 1.
In this paper, we focus on the binary classification settings. For the datasets with multiple classes (dna and satimage), we test the one-vs-rest binary classification performance, by treating data from one class as positive, and randomly selecting same amount of data from the remaining classes as negative. For each dataset, the attributes are scaled to .
Experimental Setup. Following (Rüeping, 2010), we first randomly split the data into bags of a fixed size. Bag sizes of 2, 4, 8, 16, 32, 64 are tested. We then conduct experiments with 5-fold cross validation. The performance is evaluated based on the average classification accuracy on the individual test instances. We repeat the above processes 5 times (randomly selecting negative examples for the multi-class datasets, and randomly splitting the data into bags), and report the mean accuracies with standard deviations.
The parameters are tuned by an inner cross validation loop on the training subset of each partition of the 5-fold validation. Because no instance-level labels are available during training, we use the bag-level error on the validation bags to tune the parameters:
in which and denote the predicted and the ground-truth proportions for the -th validation bag.
For MeanMap, the parameter is tuned from . For InvCal, the parameters are tuned from , and . For alter-SVM, the parameters are tuned from , and . For conv-SVM, the parameters are tuned from , and . Two kinds of kernels are considered: linear and RBF. The parameter of the RBF kernel is tuned from .
We randomly initialize alter-SVM 10 times, and pick the result with the smallest objective value. Empirically, the influence of random initialization to other algorithms is minimal.
Results. Table 2 and Table 4 show the results with linear kernel, and RBF kernel, respectively. Additional experimental results are provided in the supplementary material. Our methods consistently outperform MeanMap and InvCal, with p-value 0.05 for most of the comparisons (more than 70%). For larger bag sizes, the problem of learning from label proportions becomes more challenging due to the limited amount of supervision. For these harder cases, the gains from SVM are typically even more significant. For instance, on the dna-2 dataset, with RBF kernel and bag size 64, alter-SVM outperforms the former works by 19.82% and 12.69%, respectively (Table 4).
A Large-Scale Experiment. We also conduct a large-scale experiment on the cod-rna.t dataset containing about 271K points. The performance of InvCal and alter-SVM with linear kernel are compared. The experimental setting is the same as for the other datasets. The results in Table 3 show that alter-SVM consistently outperforms InvCal. For smaller bag sizes also, alter-SVM outperforms InvCal, though the improvement margin reduces due to sufficient amount of supervision.
In Section id1, because the bags were randomly generated, distribution of is approximately Gaussian for moderate to large . It is intuitive that the performance will depend on the distribution of proportions . If is either 0 or 1, the bags are most informative, because this leads to the standard supervised learning setting. On the other hand, if ’s are close to each other, the bags will be least informative. In fact, both MeanMap and InvCal cannot reach a numerically stable solution in such case. For MeanMap, the linear equations for solving class means will be ill-posed. For InvCal, because all the “super-instances” are assumed to have the same regression value, the result is similar to random guess.
SVM, on the other hand, can achieve good performance even in this challenging situation. For example, when using the vote dataset, with bag sizes 8 and 32, , (same as prior), with linear kernel, alter-SVM has accuracies(%) and , and conv-SVM has accuracies(%) and , respectively. These results are close to those obtained for randomly generated bags in Table 2. This indicates that our method is less sensitive to the distribution of .
Empirically, when nonlinear kernel is used, the run time of alter-SVM is longer than that of conv-SVM, because we are repeating alter-SVM multiple times to pick the solution with the smallest objective value. For instance, on a machine with 4-core 2.5GHz CPU, on the vote dataset with RBF kernel and 5-fold cross validation, the alter-SVM algorithm (repeating 10 times with the annealing loop, and one set of parameters) takes 15.0 seconds on average, while the conv-SVM algorithm takes only 4.3 seconds. But as shown in the experimental results, for many datasets, the performance of conv-SVM is marginally worse than that of alter-SVM. This can be explained by the multiple relaxations used in conv-SVM, and also the 10 time initializations of alter-SVM. As a heuristic solution for speeding up the computation, one can use conv-SVM (or InvCal) to initialize alter-SVM. For large-scale problems, in which linear SVM is used, alter-SVM is preferred, because its computational complexity is .
The speed of both alter-SVM and conv-SVM can be improved further by solving the SVM in their inner loops incrementally. For example, one can use warm start and partial active-set methods (Shilton et al., 2005). Finally, one can linearize kernels using explicit feature maps (Rahimi and Recht, 2007; Vedaldi and Zisserman, 2012), so that alter-SVM has linear complexity even for certain nonlinear kernels.
We have proposed the SVM framework for learning with label proportions, and introduced two algorithms to efficiently solve the optimization problem. Experiments on several standard and one large-scale dataset show the advantage of the proposed approach over the existing methods. The simple, yet flexible form of SVM framework naturally spans supervised, unsupervised and semi-supervised learning. Due to the usage of latent labels, SVM can also be potentially used in learning with label errors. In the future, we will design algorithms to handle bags with overlapping data. Also, we plan to investigate the theoretical conditions under which the label proportions can be preserved with the convex relaxations.
Acknowledgment. We thank Novi Quadrianto and Yu-Feng Li for their help. We thank Jun Wang, Yadong Mu and anonymous reviewers for the insightful suggestions.
- Multiple kernel learning, conic duality, and the smo algorithm. In Proceedings of the 21th International Conference on Machine learning, pp. 6. Cited by: p65.
- Alternating projections for learning with expectation constraints. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, pp. 43–50. Cited by: p17.
- Optimization techniques for semi-supervised support vector machines. The Journal of Machine Learning Research 9, pp. 203–233. Cited by: p48.
- Learning from aggregate views. In Proceedings of the 22nd International Conference on Data Engineering, pp. 3. Cited by: p18.
- Minimax theorems. Proceedings of the National Academy of Sciences of the United States of America 39 (1), pp. 42. Cited by: p67.
- Posterior sparsity in unsupervised dependency parsing. The Journal of Machine Learning Research 12, pp. 455–490. Cited by: p17.
- Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning, pp. 200–209. Cited by: p48.
- Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 217–226. Cited by: p49.
- Extension of TSVM to multi-class and hierarchical text classification problems with general losses.. In Proceeding of the 24th International Conference on Computational Linguistics, pp. 1091–1100. Cited by: p31.
- The cutting-plane method for solving convex programs. Journal of the Society for Industrial & Applied Mathematics 8 (4), pp. 703–712. Cited by: p71.
- Learning about individuals from group statistics. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence, pp. 332–339. Cited by: p18, p87.
- Semi-supervised learning using label mean. In Proceedings of the 26th International Conference on Machine Learning, pp. 633–640. Cited by: p17, p19, p31.
- Tighter and convex maximum margin clustering. In Proceeding of the 12th International Conference on Artificial Intelligence and Statistics, pp. 344–351. Cited by: p50, p58, p76, p83.
- Simple, robust, scalable semi-supervised learning via expectation regularization. In Proceedings of the 24th International Conference on Machine Learning, pp. 593–600. Cited by: p17.
- Supervised learning by training on aggregate outputs. In Proceedings of the 7th International Conference on Data Mining, pp. 252–261. Cited by: p18.
- Estimating labels from label proportions. The Journal of Machine Learning Research 10, pp. 2349–2374. Cited by: p11, p14, p18, p31, p37, p87.
- Random features for large-scale kernel machines. Advances in Neural Information Processing Systems 20, pp. 1177–1184. Cited by: p102.
- SimpleMKL. The Journal of Machine Learning Research 9, pp. 2491–2521. Cited by: p83.
- SVM classifier estimation from group probabilities. In Proceedings of the 27th International Conference on Machine Learning, pp. 911–918. Cited by: p11, p15, p31, p32, p87, p91.
- Incremental training of support vector machines. Neural Networks, IEEE Transactions on 16 (1), pp. 114–131. Cited by: p102.
- Learning from label proportions by optimizing cluster model selection. In Proceedings of the 2011 European Conference on Machine Learning and Knowledge Discovery in Databases-Volume Part III, pp. 349–364. Cited by: p19.
- Efficient additive kernels via explicit feature maps. Pattern Analysis and Machine Intelligence, IEEE Transactions on 34 (3), pp. 480–492. Cited by: p102.
- Maximum margin clustering. Advances in Neural Information Processing Systems 17, pp. 1537–1544. Cited by: footnote 2, p19, p31, p50, p58.