# Implicitly Constrained Semi-Supervised Linear Discriminant Analysis

## Abstract

Semi-supervised learning is an important and active topic of research in pattern recognition. For classification using linear discriminant analysis specifically, several semi-supervised variants have been proposed. Using any one of these methods is not guaranteed to outperform the supervised classifier which does not take the additional unlabeled data into account. In this work we compare traditional Expectation Maximization type approaches for semi-supervised linear discriminant analysis with approaches based on intrinsic constraints and propose a new principled approach for semi-supervised linear discriminant analysis, using so-called implicit constraints. We explore the relationships between these methods and consider the question if and in what sense we can expect improvement in performance over the supervised procedure. The constraint based approaches are more robust to misspecification of the model, and may outperform alternatives that make more assumptions on the data in terms of the log-likelihood of unseen objects.

## 1Introduction

In many real-world pattern recognition tasks, obtaining labeled examples to train classification algorithms is much more expensive than obtaining unlabeled examples. These tasks include document and image classification [1] where unlabeled objects can easily be downloaded from the web, part of speech tagging [2], protein function prediction [3] and many others. Using unlabeled data to improve the training of a classification procedure, however, requires semi-supervised variants of supervised classifiers to make use of this additional unlabeled data. Research into semi-supervised learning has therefore seen an increasing amount of interest in the last decade [4].

In supervised learning adding additional labeled training data improves performance for most classification routines. This does not generally hold for semi-supervised learning [5]. Adding additional unlabeled data may actually deteriorate classification performance. This can happen when the underlying assumptions of the model do not hold. In effect, disregarding the unlabeled data can lead to a better solution.

In this work we consider linear discriminant analysis (LDA) applied to classification. Several semi-supervised adaptations of this supervised procedure have been proposed. These approaches may suffer from the problem that additional unlabeled data degrade performance. To counter this problem, [6] introduced moment constrained LDA, which offers a more robust type of semi-supervised LDA. The recently introduced idea of implicitly constrained estimation [7], is another method that relies on constraints given by the unlabeled data. We compare these two approaches to other semi-supervised methods, in particular, expectation maximization and self-learning, and empirically study in what sense we can expect improvement by employing any of these semi-supervised methods.

The contributions of this work are the following:

Introduce a new, principled approach to semi-supervised LDA: implicitly constrained LDA

Offer a comparison of semi-supervised versions of linear discriminant analysis

Explore ways in which we can expect these semi-supervised methods to offer improvements over the supervised variant, in particular in terms of the log likelihood

The rest of this paper is organized as follows. After discussing related work, we introduce several approaches to semi-supervised linear discriminant analysis. These methods are then compared on an illustrative toy problem and in an empirical study using several benchmark datasets. We end with a discussion of the results and conclude.

## 2Related Work

Some of the earliest work on semi-supervised learning was done by [8] who studied the self-learning approach applied to linear discriminant analysis. This has later also been referred to as Yarowsky’s algorithm [10]. This approach is closely related to Expectation Maximization [11], where, in a generative model, the unknown labels are integrated out of the likelihood function and the resulting marginal likelihood is maximized [12]. More recent work on discriminative semi-supervised learning has focussed on introducing assumptions that relate unlabeled data to the labeled objects [4]. These assumptions usually take the form of either a manifold assumption [13], encoding that labels change smoothly in a low-dimensional manifold, or a low-density class separation assumption used in, for instance, transductive support vector machines [14] and entropy regularization [16].

Work on semi-supervised LDA has tried to incorporate unlabeled data by leveraging the increase in accuracy of estimators of quantities that do not rely on labels. An approach relying on the more accurate estimate of the total covariance matrix of both labeled and unlabeled objects is taken for dimensionality reduction in Normalized LDA, proposed by [17] and similar work by [18]. In addition to this covariance matrix, [6] also include the more accurate estimate of the overal mean of the data and propose two solutions to solve a subsequent optimization problem. Building on these results, in [7] we introduced implicitly constrained least squares classification, a semi-supervised adaptation of least squares classification. Since this procedure proved both theoretically and practically successful for a discriminative classifier, here we consider whether the idea of implicitly constrained semi-supervised learning can be extended to generative classifiers such as LDA.

## 3Methods

We will first introduce linear discriminant analysis as a supervised classification algorithm and discuss different semi-supervised procedures. We will consider 2-class classification problems, where we are given an design matrix , where is the number of labeled objects and is the number of features. For these observations we are given a label vector . Additionally, in the semi-supervised setting we have an design matrix without a corresponding for the unlabeled observations.

### 3.1Supervised LDA

In supervised linear discriminant analysis, we model the 2 classes as having multivariate normal distributions with the same covariance matrix and differing means and . To estimate the parameters of this model, we maximize the likelihood, or, equivalently, the log likelihood function:

where , denotes the density of a multivariate normal distribution with mean and covariance evaluated at and denotes the prior probability for class . The closed form solution to this maximization is given by the estimators:

Where the maximum likelihood estimator is a biased estimator for the covariance matrix. Given a set of labeled objects , we can estimate these parameters and find the posterior for a new object using:

This posterior distribution can be employed for classification by assigning objects to the class for which its posterior is highest. We now consider several adaptations of this classification procedure to the semi-supervised setting.

### 3.2Self-Learning LDA (SLLDA)

A common and straightforward adaptation of any supervised learning algorithm to the semi-supervised setting is self-learning, also known as bootstrapping or Yarowsky’s algorithm [8]. Starting out with a classifier trained on the labeled data only, labels are predicted for the unlabeled objects. These objects, with their imputed labels are then used in the next iteration to retrain the classifier. This new classifier is now used to relabel the unlabeled objects. This is done until the predicted labels on the unlabeled objects converge. [11] studies the underlying loss that this procedure minimizes and proves its convergence.

### 3.3Expectation Maximization LDA (EMLDA)

Assuming the mixture model of Equation and treating the unobserved labels as latent variables, a possible adaptation of this model is to add a term for the unlabeled data to the objective function and to integrate out the unknown labels, , to find the marginal likelihood:

Maximizing this marginal likelihood, or equivalently, the log of this function is harder then the supervised objective in Equation , since the expression contains a log over a sum. However, we can solve this optimization problem using the well-known expectation maximization (EM) algorithm [12]. In EM, the log over the sum is bounded from below through Jensen’s inequality. In the M step of the algorithm, we maximize this bound by updating the parameters using the imputed labels obtained in the E step. In practice, the M step consists of the same update as in Equation , where the sum is no longer over the labeled objects but also the unlabeled objects using the imputed posteriors, or responsibilities, from the E step. In the E step the lower bound is made tight by updating the imputed labels using the posterior under the new parameter estimates. This is done until convergence. In effect this procedure is very similar to self-learning, where instead of hard labels, a probability over labelings is used. Both self-learning and EM suffer from the problem of wrongly imputed labels that can reinforce their wrongly imputed values because the parameters are updated as if they were the true labels.

### 3.4Moment Constrained LDA (MCLDA)

An alternative to the EM-like approaches like EMLDA and SLLDA was proposed by [19] in the form of moment constrained parameter estimation. The main idea is that there are certain constraints that link parameters that are calculated using feature values alone, with parameters which require the labels. In the case of LDA [6], for instance, the overal mean of the data is linked to the means of the two classes through:

Were is the overal mean on all the data and therefore does not depend on the labels. The total covariance matrix is linked to the within-class covariance matrix and between-class covariance matrix , the covariance matrix of the means. Only the latter two rely on the labels:

Recognizing that the unlabeled data allow us to more accurately estimate the parameters in these constraints that do not rely on the labels, [6] points out that this more accurate estimate will generally violate the constraints, meaning the other label-dependent estimates should be updated accordingly.

An ad hoc way to update the parameters based on these more accurate estimates [6] leads to the following updated moment constrained estimators:

where and are the overal mean and overal covariance estimated on all labeled and unlabeled data, while is the overal covariance estimated on the labeled data alone.

Alternatively and slightly more formally, [20] forces the constraints to be satisfied by maximizing the likelihood on the labeled objects under the constraints in Equations and . This leads to a non-convex objective function that can be solved numerically. In this work we use the simpler ad hoc constraints.

### 3.5Implicitly Constrained LDA (ICLDA)

The former approach requires the identification of specific constraints. Ideally, we would like these constraints to emerge implicitly from a choice of supervised learner and a given set of unlabeled objects. Implicitly constrained semi-supervised learning attempts to do just that. The underlying intuition is that if we could enumerate all possible labelings, and train the corresponding classifiers, the classifier based on the true but unknown labels is in this set. This classifier would generally outperform the supervised classifier. Two problems arise:

How do we find a classifier in this set that is close to the one based on the true but unknown labels?

How do we efficiently traverse this enormous set of possible labelings without having to enumerate them all?

As for the first problem: a safe way to know how well a solution performs in terms of our supervised objective is to estimate its performance using the labeled objects. We therefore propose the following objective:

where

and are the design matrix and class vector extended with the unlabeled data. This can be interpreted as optimizing the same objective function as supervised LDA, with the additional constraint that the solution has to attainable by a particular assignment of responsibilities (partial assignments to classes) for the unlabeled objects.

As for the second problem: since, for a given imputed labeling, we have a closed form solution for the parameters, the gradient of the supervised loss with respect to the responsibilities can be found using

where is the function that has as input a particular labeling of the points, and outputs the parameters , similar to Equation .

This can be used to efficiently move through the set of solutions using a simple gradient ascent procedure that takes into account the bounds on the responsibilities.

## 4Experimental setup and results

We present simulations on an illustrative toy dataset and a set of benchmark datasets. Other than the classifiers covered in the previous section, we also include the LDA classifier trained using all labels of the unlabeled data (LDAoracle) as an upper bound on the performance of any semi-supervised procedure. The experiments can be reproduced using code from the authors’ website.

### 4.1Toy problems

To illustrate the behaviour of ICLDA when compared to EMLDA we consider two toy datasets. In both cases we have two multivariate normal distributions centered at respectively and and equal covariance , with the identity matrix. An example is given in Figure 1. In the bottom row, these two gaussians correspond to the different classes. In the top row, we consider the case where the decision boundary is actually perpendicular to the boundary in the other setting. This means that the bottom row corresponds exactly to the assumptions of EM, while this is not the case in the top row. Figure 1 illustrates what happens in a particular sample from this problem were we draw 10 labeled and 990 unlabeled objects. When the assumption does not hold, EMLDA forces the decision boundary to fall between the two gaussian clusters leading to a much worse solution then the supervised LDA based on only a few labeled examples. The ICLDA solution does not deviate from the correct boundary by much. When the assumptions do hold, EMLDA finds the correct boundary, as expected, while ICLDA only does minor adjustments in this case.

While one could claim that ICLDA is more robust, one could also expect ICLDA to never lead to any improvements. Figure 2 shows the results when resampling from the data distribution in the second example and shows that ICLDA does lead to improvement on average in the second dataset, while not making the mistake in the first dataset where the LDA assumptions do not hold.

### 4.2Simulations on Benchmark datasets

We test the behaviour of the considered procedures using datasets from the UCI machine learning repository [21], as well as from [4]. Their characteristics can be found in Table 1.

Name | # Objects | #Features | Source |
---|---|---|---|

Haberman | 305 | 4 | |

Ionosphere | 351 | 33 | |

Parkinsons | 195 | 20 | |

Pima | 768 | 9 | |

Sonar | 208 | 61 | |

SPECT | 265 | 23 | |

SPECTF | 265 | 45 | |

Transfusion | 748 | 4 | |

WDBC | 568 | 30 | |

BCI | 400 | 118 | |

A cross-validation experiment was carried out as follows. Each of the datasets were split into folds. Every fold was used as a validation set once, while the other nine folds were used for training. The data in these nine folds was randomly split into a labeled and an unlabeled part, where the labeled part had objects, while the rest was used as unlabeled objects. This procedure was repeated times and the average error and average negative log likelihood (the loss function of LDA) on the test set was determined. The results can be found in Tables Table 2 and Table 3.

To study the behaviour of these classifiers for differing amount of unlabeled data, we estimated semi-supervised learning curves by randomly drawing labeled objects from the datasets, and using an increasing randomly chosen set as unlabeled data. The remaining objects formed the test set. This procedure was repeated 500 times and the average and standard error of the classification error and negative log likelihood were determined. The learning curves for 3 datasets can be found in Figure 3.

Dataset | LDA | LDAoracle | MCLDA | EMLDA | SLLDA | ICLDA |
---|---|---|---|---|---|---|

Haberman | ||||||

Ionosphere | ||||||

Parkinsons | ||||||

Pima | ||||||

Sonar | ||||||

SPECT | ||||||

SPECTF | ||||||

Transfusion | ||||||

WDBC | ||||||

BCI | ||||||

Dataset | LDA | LDAoracle | MCLDA | EMLDA | SLLDA | ICLDA |
---|---|---|---|---|---|---|

Haberman | ||||||

Ionosphere | ||||||

Parkinsons | ||||||

Pima | ||||||

Sonar | ||||||

SPECT | ||||||

SPECTF | ||||||

Transfusion | ||||||

WDBC | ||||||

BCI | ||||||

We find that overal in terms of error rates (Table 2), MCLDA seems to perform best, being both more robust than the EM approaches as well as effective in using the unlabeled information in improving error rates. While ICLDA is robust and has the best performance on 2 of the datasets, it is conservative in that it does not show improvements in terms of the classification error for many datasets where the other classifiers do offer improvements.

The picture is markedly different when we consider the log likelihood criterion that supervised LDA optimizes, evaluated on the test set (Table 3). Here ICLDA outperforms all other methods on the majority of the datasets.

## 5Discussion

The results show that ICLDA provides the safest option among the semi-supervised approaches to LDA that we considered. At least in terms of the log-likelihood, it provides the best performance by far. A particularly interesting observation is that the implicit constraints, in many a case, seem to add a lot to the constraints that MCLDA enforces, as also the latter classifier is consistently outperformed by ICLDA in terms of the log likelihoods achieved on the test set. As yet, we have no full understanding in what way the implicit constraints add restrictions beyond the moment constraints, apart from the fact that the former are stricter than the latter. A deeper insight into this issue might cast further light on the workings of ICLDA.

While it is the safest version, ICLDA may be *too* safe, in that it does not attain performance improvement in terms of classification error in many cases where MCLDA or the EM approaches do offer improvements. In terms of the loss on the test set, however, ICLDA is the best performing method. Since this is the objective minimized by supervised LDA as well, perhaps this is the best we could hope for in a true semi-supervised adaptation of LDA. We found similar empirical and theoretical performance results in terms of improvements in the loss on the test set when applying the implicitly constrained framework to the least squares classifier [7]. How then, this improvement in “surrogate” loss relates to the eventual goal of classification error, is unclear, especially for a non-margin based loss function such as the negative log likelihood [22]. However, since ICLDA does offer the best behaviour of supervised LDA’s loss on the test set, ICLDA could be considered a step towards a principled semi-supervised version of LDA.

An open question regarding the objective function associated with ICLDA is to what extent it is convex. The solution in terms of the responsibility vector is non-unique: different labelings of the points can lead to the same parameters. In terms of the parameters, however, the optimization seems to converge to a unique global optimum. While we do not have a formal proof of this, as in the case of implicitly constrained least squares classification, we conjecture that the objective function is convex in the parameters, at least in the case in which we choose to parameterize LDA by means of its canonical parameters [23]. In this case, LDA does lead to a convex optimization problem.

We find that the behaviour of EMLDA is more erratic than that of SLLDA. The hard label assignments could have a regularizing effect on the semi-supervised solutions, making self-learning a good and fast alternative to self-learning. Note that safer versions of SLLDA and EMLDA could be obtained by introducing a weight parameter to control the influence of the unlabeled data [8]. In the limited labeled data setting, it is hard to correctly set this parameter. While this may help when dealing with larger sample sizes, the constraint approaches bring us closer to methods that always perform at least as well as their supervised counterpart.

## 6Conclusion

ICLDA is a principled and robust adaptation of LDA to the semi-supervised setting. In terms of error rates, it may be overly conservative. When measured in terms of the loss on the test set, however, it outperforms other semi-supervised methods. It therefore seems that there are opportunities for robust semi-supervised learners, although the performance criterion that we should consider may not be the error rate, but rather the loss that the supervised learner minimizes [24].

## Acknowledgment

This work was partly funded by project P24 of the Dutch public/private research network COMMIT.

### References

- =2plus 43minus 4 K. Nigam, A. K. McCallum, S. Thrun, and T. Mitchell, “Text classification from labeled and unlabeled documents using EM,”
*Machine learning*, vol. 34, pp. 1–34, 2000. =0pt - =2plus 43minus 4 D. Elworthy, “Does Baum-Welch re-estimation help taggers?” in
*Proceedings of the fourth conference on Applied natural language processing*, 1994, pp. 53–58. =0pt - =2plus 43minus 4 J. Weston, C. Leslie, E. Ie, D. Zhou, A. Elisseeff, and W. S. Noble, “Semi-supervised protein classification using cluster kernels.”
*Bioinformatics*, vol. 21, no. 15, pp. 3241–7, Aug. 2005. =0pt - O. Chapelle, B. Schölkopf, and A. Zien,
*Semi-supervised learning*.1em plus 0.5em minus 0.4emMIT press, 2006. - F. Cozman and I. Cohen, “Risks of Semi-Supervised Learning,” in
*Semi-Supervised Learning*, O. Chapelle, B. Schölkopf, and A. Zien, Eds.1em plus 0.5em minus 0.4emMIT press, 2006, ch. 4, pp. 56–72. - =2plus 43minus 4 M. Loog, “Semi-supervised linear discriminant analysis through moment-constraint parameter estimation,”
*Pattern Recognition Letters*, vol. 37, pp. 24–31, Mar. 2014. =0pt - =2plus 43minus 4 J. H. Krijthe and M. Loog, “Implicitly Constrained Semi-Supervised Least Squares Classification,” Tech. Rep., 2013. =0pt
- G. J. McLachlan, “Iterative Reclassification Procedure for Constructing an Asymptotically Optimal Rule of Allocation in Discriminant Analysis,”
*Journal of the American Statistical Association*, vol. 70, no. 350, pp. 365–369, 1975. - P. Taylor and G. J. Mclachlan, “Estimating the Linear Discriminant Function from Initial Samples Containing a Small Number of Unclassified Observations,”
*Journal of the American Statistical Association*, vol. 72, no. 358, pp. 403–406, 1977. - =2plus 43minus 4 D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,”
*Proceedings of the 33rd annual meeting on Association for Computational Linguistics*, pp. 189–196, 1995. =0pt - =2plus 43minus 4 S. Abney, “Understanding the yarowsky algorithm,”
*Computational Linguistics*, vol. 30, no. 3, pp. 365–395, 2004. =0pt - =2plus 43minus 4 A. Dempster, N. Laird, and D. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,”
*Journal of the Royal Statistical Society. Series B*, vol. 39, no. 1, pp. 1–38, 1977. =0pt - =2plus 43minus 4 X. Zhu, Z. Ghahramani, and J. Lafferty, “Semi-supervised learning using gaussian fields and harmonic functions,” in
*Proceedings of the 20th International Conference on Machine Learning*, 2003, pp. 912–919. =0pt - K. P. Bennett and A. Demiriz, “Semi-supervised support vector machines,” in
*Advances in Neural Information Processing Systems 11*, 1998. - T. Joachims, “Transductive inference for text classification using support vector machines,” in
*Proceedings of the 16th International Conference on Machine Learning*.1em plus 0.5em minus 0.4emMorgan Kaufmann Publishers, 1999, pp. 200–209. - =2plus 43minus 4 Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in
*Advances in Neural Information Processing Systems 17*, L. K. Saul, Y. Weiss, and L. Bottou, Eds.1em plus 0.5em minus 0.4emCambridge, MA: MIT Press, 2005, pp. 529–536. =0pt - =2plus 43minus 4 B. Fan, Z. Lei, and S. Z. Li, “Normalized LDA for semi-supervised learning,” in
*8th IEEE International Conference on Automatic Face & Gesture Recognition*, Sept. 2008, pp. 1–6. =0pt - =2plus 43minus 4 D. Cai, X. He, and J. Han, “Semi-supervised Discriminant Analysis,” in
*IEEE 11th International Conference on Computer Vision*, 2007, pp. 1–7. =0pt - =2plus 43minus 4 M. Loog, “Constrained Parameter Estimation for Semi-Supervised Learning: The Case of the Nearest Mean Classifier,” in
*Proceedings of the 2010 European Conference on Machine learning and Knowledge Discovery in Databases*, 2010, pp. 291–304. =0pt - =2plus 43minus 4 M. Loog and A. C. Jensen, “Constrained log-likelihood-based semi-supervised linear discriminant analysis,” in
*Structural, Syntactic, and Statistical Pattern Recognition*, 2012, pp. 327–335. =0pt - =2plus 43minus 4 K. Bache and M. Lichman, “{UCI} Machine Learning Repository,” 2013. =0pt
- P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe, “Convexity, Classification, and Risk Bounds,”
*Journal of the American Statistical Association*, vol. 101, no. 473, pp. 138–156, Mar. 2006. - E. L. Lehmann and G. Casella,
*Theory of Point Estimation*.1em plus 0.5em minus 0.4emSpringer-Verlag, 1998. - M. Loog and A. C. Jensen, “Semi-Supervised Nearest Mean Classification through a constrained Log-Likelihood,”
*IEEE Transactions on Neural Networks and Learning Systems*, 2014, conditionally accepted.