Learning from a Teacher using Unlabeled Data
Knowledge distillation is a widely used technique for model compression. We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset. We empirically show that a teacher model can transfer this knowledge to a student model even on an out-of-distribution dataset. Using this approach, we show promising results on MNIST, CIFAR-10, and Caltech-256 datasets using unlabeled image data from different sources. Our results are encouraging and help shed further light from the perspective of understanding knowledge distillation and utilizing unlabeled data to improve model quality.
Knowledge distillation was introduced by Hinton et al. (Hinton14, ) primarily for model compression by introducing a method to replicate the performance of model ensembles with a single model. One or more teacher models can be used to distill information into a single student model, generally of lower capacity in terms of model parameters. The teacher models help by providing soft-labels to the student, which allow it to learn a function to better discriminate between the different classes (Figure 1).
Our core hypothesis is that a teacher model can help the student model discriminate between the classes using not just the original training data, but also through a large corpus of unlabeled data available from a different distribution.
To understand the intuition and ground our hypothesis in a real-word example, let us consider the task of labeling an image of a dog into one of hundred or so breeds. We can use a distillation setup with teacher and student models to learn an image classifier model. Given an unlabeled dataset of all animals excluding dogs, can a teacher model help the student distinguish that certain animals look closer to certain dog / breeds? As an example, an Arctic Wolf might look similar to a Husky / Alaskan Malamute. While, a lot of other animals might match some selective attributes of specific breeds (e.g., size, fur texture, color, etc.), others might not match any characteristics, and hence serve as hard negatives.
If the teacher model is resilient to out-of-distribution data, and can be useful for a student model, it will have several applications such as:
Using readily available unlabeled data for improving model quality, as a complementary step to the standard supervised training.
Checking the robustness of a model to being replicated, when its predictions exposed through Cloud or a similar API.
In this paper, we propose a method for testing our hypothesis and demonstrate results via the following:
Explore training student model(s) in an unsupervised setting, on an unlabeled dataset () which is completely different from the original labeled dataset () used to train the teacher model.
Demonstrate via experimental results on multiple datasets and also compare it to training directly on the original labeled dataset.
Show visualizations of embeddings of examples from both the labeled and unlabeled datasets, using the teacher model which was directly trained on the labeled dataset.
2 Related Work
It seems that the first approach towards transferring knowledge between networks was introduced by Caruna et al. Caruana06 , but the work was limited to shallow networks. Knowledge Distillation introduced by Hinton et al. Hinton14 allowed compressing wide and deep ensemble networks by training a smaller network that mimicked the prediction of the bigger ensemble.
Several papers point out the applications of Knowledge Distillation towards learning compact models. Chen et al. Chen17 use it to train smaller Object Detection models. Luo et al. Luo16 use it to optimize Face Recognition models.
There is existing literature in the area of working with different data distributions for training a model. Abbasi et al. Abbasi18 explore that out-distribution samples can be used to augment a convolutional work to make it more resistant to adversarial attacks. Wang19 uses Distillation, but to construct a dataset that can be used to fine-tune a model trained on a different dataset, to predict examples based on the labeled dataset. Motiian17 provide a few-shot method for domain translation based on adversarial techniques.
Radosavovic et al. Radosavovic18 ensemble predictions over several unlabeled images during training (and optionally inference) to improve the accuracy of object detection models, but they use data from the same distribution which is the COCO labeled and unlabeled datasets. In contrast, our method does not require ensembling of transformed examples (or any other usecase-specific transforms), and we demonstrate our results on unlabeled data obtained from different sources and in some cases, with a very visually recognizable different distribution.
The original training and test datasets are referred to as the labeled training and test datasets. Let us denote them as (, ) and (, ) respectively, where refers to the input examples, and refers to the corresponding labels.
We also assume that we have access to an ideally large unlabeled dataset, which belongs to the same modality as our labeled dataset and drawn from a related, but not necessarily the same distribution. Even if this dataset does have labels, we would ignore them. Let us denote this dataset as .
3.1 Blind Distillation
Next, we describe our method for training a teacher/student combination for the proposed distillation scenario (see Figure LABEL:blind-dist for the flowchart).
Train a teacher model , using (, ) as training data, and (, ) as test data.
Freeze the teacher model, and let return the logits of the teacher model for the input .
Train a student model :
Compute the logits of on . .
Similarly, compute the logits of on . .
Minimize the cross-entropy loss, .
Repeat (a)-(c) until convergence.
Compute the accuracy of the model using (, ).
Note that the student model trained with this method has not seen any examples from the labeled set during training. The only source of information it has about the distribution of is through the teacher model , and the information the teacher model can extract from . Hence, we refer to this method as Blind Distillation.
3.2 Fine-Tune with Labeled Data
The unlabeled data being used in the Blind Distillation phase will very likely have a different distribution as compared to the labeled data. Since the student model is finally evaluated on the labeled data, we can improve its using a part of the original labeled dataset since this is available.
Train the student model , with the Blind Distillation method above.
Create a subset of the labeled training dataset with samples per class . Let us denote this dataset as (, ).
Fine tune the model by distilling from the teacher model using examples from . Let us refer to the model trained by this fine-tuning as .
During our experiments, we found that fine-tuning with a low learning rate (when compared to the Blind Distillation phase) with a steady decay, helps keep the student model’s loss within a reasonable range. This is intuitive because the student model has been trained to expect examples from the unlabeled dataset’s distribution. Introducing examples from the labeled distribution without a careful learning rate schedule would suddenly lead to a large gradient flow.
We experimented with our method on standard image classification tasks. In all of our experiments, we set the distillation temperature, .
We used the following datasets for our experiments:
MNIST: MNIST MNIST is a popular dataset of handwritten digits, used as a benchmark in image classification tasks.
Fashion-MNIST: Fashion-MNIST is a dataset of fashion articles that is intended to be a drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms FashionMNIST . Each example in the dataset is a grayscale image of size 28x28, and belongs to one of 10 classes (the same as MNIST), however the classes are completely different between the two datasets.
CIFAR-10 & CIFAR-100: The CIFAR-10 dataset consists of 60000 32x32 colour images belonging to 10 classes. CIFAR-100 is similar to CIFAR-10, except that each image belongs to one of 100 classes. CIFAR-100 can be considered similar to CIFAR-10 in the taxonomy that they represent CIFARDatasets . However, none of the classes in CIFAR-10 overlap with any class in CIFAR-100.
Caltech-256: The CalTech-256 dataset is a collection of 30,000 images spread across 256 classes Caltech256Dataset . Each image belongs to one of the 256 classes.
OpenImages: Open Images OpenImagesDataset has 9M images spanning 20,000 classes. The images often contain complex scenes with several objects.
ImageNet: ImageNet ImageNetDataset is a popular image classification dataset having 1M images spanning 1000 classes. Each image contains one image.
We used Google Colab Colab for experimenting with MNIST, Fashion MNIST, and CIFAR datasets, using one GPU at a time. For experimenting with Caltech-256 and OpenImages we used an internal Machine Learning training platform using 4 GPUs at a time.
In the following subsection, we have broken down the experimentation section based on the labeled and unlabeled dataset pairs.
4.3 MNIST Classification Results: <Mnist, Fashion MNIST>
We experiment with two architectures:
CNN: Two convolutional layers, with batch-normalization and max-pooling layers between them, and one fully connected layer. Total: 1.12M params.
MLP: Three fully connected layers. Total: 85,000 params.
Table 1 shows the accuracy of the teacher model (column 1), as compared to the accuracy of the student model when blind-distilled using Fashion MNIST and the teacher model (column 2). The third column shows the accuracy when the student model is fine-tuned using the MNIST labeled data.
When using CNN-based teacher and student models, the student gets 96.2% accuracy when compared to the teacher’s 99.5% accuracy, without using any MNIST examples.
Similarly using MLP-based teacher and student models, the student gets 84% accuracy when compared to the teacher’s 97.69% accuracy, again without using any MNIST examples.
We will further discuss the performance differences between the combination of the two model architectures in the Discussion section.
Another interesting observation is that when the teacher and student model capacities are lower, it leads to lower student model accuracy. This implies that the student is learning a non-trivial function, and both the teacher model accuracy as well as the student model’s own capacity to learn are important.
|Teacher Model||Blind Distillation||Blind Distillation|
|Accuracy ()||Accuracy ()||& Fine-Tune ()|
|99.5% (CNN)||96.2% (CNN)||99.6%|
|97.69% (MLP)||84% (MLP)||97.95%|
|99.5% (CNN)||68.60% (MLP)||98.16%|
|97.69% (MLP)||89.45% (CNN)||99.26%|
4.4 CIFAR Image Classification Results: <Cifar-10, Cifar-100>
For these experiments, we use the same model architecture for both the teacher and the student.
The architecture for both the teacher and the student is a simple Convolutional Neural Net with 6 convolutional layers, and one feed forward layer. We add batch-normalization layers and dropout layers to help improve accuracy.
The student model reaches an accuracy of 79.5%, without having seen any examples from the CIFAR-10 dataset. Fine-tuning with the labeled dataset improved the accuracy to 86.37%, which was better than the teacher model’s 84.73%.
|Teacher Model||Blind Distillation||Blind Distillation|
|Accuracy ()||Accuracy ()||& Fine-Tune ()|
|84.73% (CNN)||79.5% (CNN)||86.37%|
4.5 Image Classification Results with More Classes: <CalTech-256, OpenImages>
In this experiment, both the teacher and student models were based on the MobileNet v2 architecture MobileNetV2 .
This is a more challenging experiment, as Caltech-256 is an ImageNet like dataset with a large number of classes. OpenImages and covers a wide range of concepts, with 8 different classes present on an average in the same image OpenImagesDataset . We believe that a well trained teacher model would be able to extract
Note that fine-tuning with just 10 and 20 samples / class on the CalTech256 dataset improved the accuracy of the student model from 64.0% to 71.6% and 74.46% respectively, doing significantly better than the original teacher model. Using the entire dataset to fine-tune the student model leads to an even bigger jump to 76.49% accuracy.
|Teacher Model||Blind Distillation||Blind Distillation &|
|Accuracy ()||Accuracy ()||Fine-Tune Accuracy ()|
4.6 Image Classification Results with Even More Classes: <ImageNet, OpenImages>
In this experiment as well, both the teacher and student models were based on the MobileNet v2 architecture MobileNetV2 .
|Teacher Model||Blind Distillation||Blind Distillation &|
|Accuracy ()||Accuracy ()||Fine-Tune Accuracy ()|
In this section we further analyse the results of our experiments, w.r.t. the embeddings of the labeled and unlabeled data, effect of teacher and student model architecture, etc.
5.1 Joint embedding of examples from and
Figure 6 is the t-SNE plot of embeddings of MNIST and Fashion MNIST images (1000 images from either dataset) produced by the teacher model. The shades of green and blue are images belonging to Fashion MNIST. Shades of red, pink and purple belong to MNIST. Varying the t-SNE learning rate in the [100.0, 300.0] range and perplexity in the [20.0, 50.0] range gave similar plots.
We can see distinct clusters of MNIST classes, as the teacher model learnt the manifold that the MNIST images occupy.
We can also observe that the Fashion MNIST images are not spread randomly, although certain MNIST images are clearly on the Fashion MNIST side of the boundary. Certain clusters for example the bright and dark green dots are closer to some MNIST image clusters. Similarly the navy blue cluster at the top is closer to the dark red MNIST cluster. Examples drawn from these Fashion MNIST classes would therefore make the teacher predict the examples to be more likely to belong to the MNIST classes they are close to, than to the classes they are far apart from.
5.2 Student Model Architecture
In the MNIST & Fashion MNIST experiment, we experiment with two different model architectures. A simple convolutional neural net (CNN) architecture outperformed a multi-layer perceptron (MLP) network. This was intuitive, and expected. All other things held constant, we find that if the student network had more capacity to learn, it performed better in our setup.
5.3 Teacher Model Architecture
We see an interesting phenomenon in the Blind Distillation phase when changing the teacher model architecture while keeping the student model architecture same. If the teacher was an MLP, and the student was a CNN, it was less accurate than was when the teacher was a CNN as well. Our hypothesis is that a CNN teacher model architecture extracted features that were different from what an MLP extracted, hence leading to a dip in the performance. Similarly, when the teacher was a CNN, but the student was an MLP, it performed quite worse than when the teacher was an MLP too.
However, when these blind-distilled models were fine-tuned on the actual datasets, we found that both the performance dips were resolved. When the MLP model was being fine-tuned by a CNN model, its performance improved to 98.16%, which is better than either vanilla supervised training or blind-distillation followed by fine-tuning with an MLP.
Similarly, the performance of the CNN model being fine-tuned by an MLP improved its accuracy from 89.45% to 99.26%. This is lower than the CNN model being trained by itself / distilled from another CNN model, but this is expected since the teacher model has 10x less parameters.
5.4 Accuracy of the Teacher
It is intuitive to expect a more accurate teacher to lead to a better trained student, and we saw that pattern across the different datasets. In addition, we saw that while keeping the teacher model architectures the same, the cross entropy loss at convergence was higher for the student distilled from the more accurate teacher model. This hints at the case that more accurate teachers will teach more complex functions.
5.5 Comparison with Transfer Learning
Transfer Learning can transfer knowledge between labeled domains. Our work demonstrates the efficacy of distillation when used in an unlabeled domain.
As such, our methods are complementary to Transfer Learning, since the Teacher and/or the student can be initialized with a transfer-learnt checkpoint.
In this paper, we saw that a teacher model used in a distillation setup, captured relationships between classes, even outside the original dataset. For example, while the MNIST and Fashion MNIST datasets look very different, using Blind Distillation, we were able to train a model to 96% accuracy, which hasn’t seen any MNIST images. Using the t-SNE plots between MNIST and Fashion MNIST images, we visually demonstrated why distillation is surprisingly effective in such a constrained setting.
Similarly, the Caltech-256 and the OpenImages dataset differ a lot in terms of the number of classes and distribution of images. Blind Distillation by itself brings us to 64.0% accuracy as compared to the 68.20% accuracy of the teacher. Fine-tuning using the labeled examples leads to a 8.30% absolute accuracy improvement. This is promising because it is relatively easier and cheaper to collect large amounts of unlabeled data, than it is to get the same amount of labeled data. Leveraging the readily available unlabeled data in conjunction with existing supervised learning on the labeled data, offers a scenario which is best of both the worlds.
From the perspective of model security research, we demonstrate that given access to a model’s logits, it is easy to replicate the given model’s performance without access to the original data. This is a problem worth investigating in its own right.
With this paper, we hope we can further the understanding of Knowledge Distillation, and inspire its applications which leverage unlabeled data to improve model quality.
-  G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network, NeurIPS Deep Learning Workshop 2014.
-  C. Bucilua, R. Caruana, A. Niculescu-Mizil. Model compression. KDD 2006.
-  G. Chen, W. Choi, X. Yu, T. Han, M. Chandraker. Learning Efficient Object Detection Models with Knowledge Distillation. NeurIPS 2017.
-  Y. Cheng, D. Wang, P. Zhou. A Survey of Model Compression and Acceleration for Deep Neural Networks. IEEE Signal Processing Magazine Special Issue on Deep Learning for Image Understanding, 2019.
-  P. Luo, Z. Zhu, Z. Liu, X. Wang, X. Tang. Face Model Compression by Distilling Knowledge from Neurons. AAAI 2016.
-  M. Abbasi, C. Gagne. Out-distribution Training Confers Robustness To Deep Neural Networks. ICLR Workshop Track 2018.
-  T. Wang, J. Zhu, A. Torralba, A. A. Efros. Dataset Distillation. arXiv:1811.10959, 2019.
-  S. Motiian, Q. Jones, S. M. Iranmanesh, G. Doretto. Few-Shot Adversarial Domain Adaptation. NeurIPS 2017.
-  I. Radosavovic, P. Dollar, R. Girshick, G. Gkioxari, K. He. Data Distillation: Towards Omni-Supervised Learning. CVPR 2018.
-  Y. LeCun, C. Cortes, C. J. C. Burges. The MNIST Dataset Of Handwritten Digits.
-  H. Xiao, K. Rasul, R. Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv:1708.07747, 2017.
-  A. Krizhevsky. https://www.cs.toronto.edu/~kriz/cifar.html.
-  G. Griffin, A. D. Holub, P. Perona. The Caltech 256. Caltech Technical Report.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei,. ImageNet: a large-scale hierarchical image database. CVPR, 2009.
-  Google Colaboratory. https://colab.research.google.com.
-  A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982, 2018.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei, L. ImageNet: a large-scale hierarchical image database. CVPR, 2009.
-  M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L. Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018.
-  The Fashionable “Hello World” of Deep Learning. https://medium.com/analytics-vidhya/the-fashionable-hello-world-of-deep-learning-8b9e3d60a37c