Semi-Supervised Sequence Modeling with Cross-View Training
Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.111Code is available at https://github.com/tensorflow/models/tree/master/research/cvt_text
Kevin Clark Minh-Thang Luong Christopher D. Manning Quoc V. Le Computer Science Department, Stanford University Google Brain email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org
Deep learning models work best when trained on large amounts of labeled data. However, acquiring labels is costly, motivating the need for effective semi-supervised learning techniques that leverage unlabeled examples. A widely successful semi-supervised learning strategy for neural NLP is pre-training word vectors (Mikolov et al., 2013). More recent work trains a Bi-LSTM sentence encoder to do language modeling and then incorporates its context-sensitive representations into supervised models (Dai and Le, 2015; Peters et al., 2018). Such pre-training methods perform unsupervised representation learning on a large corpus of unlabeled data followed by supervised training.
A key disadvantage of pre-training is that the first representation learning phase does not take advantage of labeled data – the model attempts to learn generally effective representations rather than ones that are targeted towards a particular task. Older semi-supervised learning algorithms like self-training do not suffer from this problem because they continually learn about a task on a mix of labeled and unlabeled data. Self-training has historically been effective for NLP (Yarowsky, 1995; McClosky et al., 2006), but is less commonly used with neural models. This paper presents Cross-View Training (CVT), a new self-training algorithm that works well for neural sequence models.
In self-training, the model learns as normal on labeled examples. On unlabeled examples, the model acts as both a âteacherâ that makes predictions about the examples and a âstudentâ that is trained on those predictions. Although this process has shown value for some tasks, it is somewhat tautological: the model already produces the predictions it is being trained on. Recent research on computer vision addresses this by adding noise to the student’s input, training the model so it is robust to input perturbations (Sajjadi et al., 2016; Wei et al., 2018). However, applying noise is difficult for discrete inputs like text.
As a solution, we take inspiration from multi-view learning (Blum and Mitchell, 1998; Xu et al., 2013) and train the model to produce consistent predictions across different views of the input. Instead of only training the full model as a student, CVT adds auxiliary prediction modules – neural networks that transform vector representations into predictions – to the model and also trains them as students. The input to each student prediction module is a subset of the model’s intermediate representations corresponding to a restricted view of the input example. For example, one auxiliary prediction module for sequence tagging is attached to only the “forward” LSTM in the model’s first Bi-LSTM layer, so it makes predictions without seeing any tokens to the right of the current one.
CVT works by improving the model’s representation learning. The auxiliary prediction modules can learn from the full model’s predictions because the full model has a better, unrestricted view of the input. As the auxiliary modules learn to make accurate predictions despite their restricted views of the input, they improve the quality of the representations they are built on top of. This in turn improves the full model, which uses the same shared representations. In short, our method combines the idea of representation learning on unlabeled data with classic self-training.
CVT can be applied to a variety of tasks and neural architectures, but we focus on sequence modeling tasks where the prediction modules are attached to a shared Bi-LSTM encoder. We propose auxiliary prediction modules that work well for sequence taggers, graph-based dependency parsers, and sequence-to-sequence models. We evaluate our approach on English dependency parsing, combinatory categorial grammar supertagging, named entity recognition, part-of-speech tagging, and text chunking, as well as English to Vietnamese machine translation. CVT improves over previously published results on all these tasks. Furthermore, CVT can easily and effectively be combined with multi-task learning: we just add additional prediction modules for the different tasks on top of the shared Bi-LSTM encoder. Training a unified model to jointly perform all of the tasks except machine translation improves results (outperforming a multi-task ELMo model) while decreasing the total training time.
2 Cross-View Training
We first present Cross-View Training and describe how it can be combined effectively with multi-task learning. See Figure 1 for an overview of the training method.
Let represent a labeled dataset and represent an unlabeled dataset We use to denote the output distribution over classes produced by the model with parameters on input . During CVT, the model alternates learning on a minibatch of labeled examples and learning on a minibatch of unlabeled examples. For labeled examples, CVT uses standard cross-entropy loss:
CVT adds auxiliary prediction modules to the model, which are used when learning on unlabeled examples. A prediction module is usually a small neural network (e.g., a hidden layer followed by a softmax layer). Each one takes as input an intermediate representation produced by the model (e.g., the outputs of one of the LSTMs in a Bi-LSTM model). It outputs a distribution over labels . Each is chosen such that it only uses a part of the input ; the particular choice can depend on the task and model architecture. We propose variants for several tasks in Section 3. The auxiliary prediction modules are only used during training; the test-time prediction come from the primary prediction module that produces .
On an unlabeled example, the model first produces soft targets by performing inference. CVT trains the auxiliary prediction modules to match the primary prediction module on the unlabeled data by minimizing
where is a distance function between probability distributions (we use KL divergence). We hold the primary module’s prediction fixed during training (i.e., we do not back-propagate through it) so the auxiliary modules learn to imitate the primary one, but not vice versa. CVT works by enhancing the model’s representation learning. As the auxiliary modules train, the representations they take as input improve so they are useful for making predictions even when some of the model’s inputs are not available. This in turn improves the primary prediction module, which is built on top of the same shared representations.
We combine the supervised and CVT losses into the total loss, , and minimize it with stochastic gradient descent. In particular, we alternate minimizing over a minibatch of labeled examples and minimizing over a minibatch of unlabeled examples.
For most neural networks, adding a few additional prediction modules is computationally cheap compared to the portion of the model building up representations (such as an RNN or CNN). Therefore our method contributes little overhead to training time over other self-training approaches for most tasks. CVT does not change inference time or the number of parameters in the fully-trained model because the auxiliary prediction modules are only used during training.
2.2 Combining CVT with Multi-Task Learning
CVT can easily be combined with multi-task learning by adding additional prediction modules for the other tasks on top of the shared Bi-LSTM encoder. During supervised learning, we randomly select a task and then update using a minibatch of labeled data for that task. When learning on the unlabeled data, we optimize jointly across all tasks at once, first running inference with all the primary prediction modules and then learning from the predictions with all the auxiliary prediction modules. As before, the model alternates training on minibatches of labeled and unlabeled examples.
Examples labeled across many tasks are useful for multi-task systems to learn from, but most datasets are only labeled with one task. A benefit of multi-task CVT is that the model creates (artificial) all-tasks-labeled examples from unlabeled data. This significantly improves the model’s data efficiency and training time. Since running prediction modules is computationally cheap, computing is not much slower for many tasks than it is for a single one. However, we find the all-tasks-labeled examples substantially speed up model convergence. For example, our model trained on six tasks takes about three times as long to converge as the average model trained on one task, a 50% decrease in total training time.
3 Cross-View Training Models
CVT relies on auxiliary prediction modules that have restricted views of the input. In this section, we describe specific constructions of the auxiliary prediction modules that are effective for sequence tagging, dependency parsing, and sequence-to-sequence learning.
3.1 Bi-LSTM Sentence Encoder
All of our models use a two-layer CNN-BiLSTM (Chiu and Nichols, 2016; Ma and Hovy, 2016) sentence encoder. It takes as input a sequence of words . First, each word is represented as the sum of an embedding vector and the output of a character-level Convolutional Neural Network, resulting in a sequence of vectors . The encoder applies a two-layer bidirectional LSTM (Graves and Schmidhuber, 2005) to these representations. The first layer runs a Long Short-Term Memory unit (Hochreiter and Schmidhuber, 1997) in the forward direction (taking as input at each step ) and the backward direction (taking at each step) to produce vector sequences and . The output of the Bi-LSTM is the concatenation of these vectors: The second Bi-LSTM layer works the same, producing outputs , except it takes as input instead of .
3.2 CVT for Sequence Tagging
In sequence tagging, each token has a corresponding label . The primary prediction module for sequence tagging produces a probability distribution over classes for the label using a one-hidden-layer neural network applied to the corresponding encoder outputs:
The auxiliary prediction modules take and , the outputs of the forward and backward LSTMs in the first222Modules taking inputs from the second Bi-LSTM layer would not have restricted views because information about the whole sentence gets propagated through the first layer. Bi-LSTM layer, as inputs. We add the following four auxiliary prediction modules to the model (see Figure 2):
The “forward” module makes each prediction without seeing the right context of the current token. The “future” module makes each prediction without the right context or the current token itself. Therefore it works like a neural language model that, instead of predicting which token comes next in the sequence, predicts which class of token comes next. The “backward” and “past” modules are analogous.
3.3 CVT for Dependency Parsing
In a dependency parse, words in a sentence are treated as nodes in a graph. Typed directed edges connect the words, forming a tree structure describing the syntactic structure of the sentence. In particular, each word in a sentence receives exactly one in-going edge going from word (called the “head”) to it (the “dependent”) of type (the “relation”). We use a graph-based dependency parser similar to the one from Dozat and Manning (2017). This treats dependency parsing as a classification task where the goal is to predict which in-going edge connects to each word .
First, the representations produced by the encoder for the candidate head and dependent are passed through separate hidden layers. A bilinear classifier applied to these representations produces a score for each candidate edge. Lastly, these scores are passed through a softmax layer to produce probabilities. Mathematically, the probability of an edge is given as:
where is the scoring function:
The bilinear classifier uses a weight matrix specific to the candidate relation as well as a weight matrix shared across all relations. Note that unlike in most prior work, our dependency parser only takes words as inputs, not words and part-of-speech tags.
We add four auxiliary prediction modules to our model for cross-view training:
Each one has some missing context (not seeing either the preceding or following words) for the candidate head and candidate dependent.
3.4 CVT for Sequence-to-Sequence Learning
We use an encoder-decoder sequence-to-sequence model with attention (Sutskever et al., 2014; Bahdanau et al., 2015). Each example consists of an input (source) sequence and output (target) sequence . The encoder’s representations are passed into an LSTM decoder using a bilinear attention mechanism (Luong et al., 2015). In particular, at each time step the decoder computes an attention distribution over source sequence hidden states as where is the decoder’s current hidden state. The source hidden states weighted by the attention distribution form a context vector: . Next, the context vector and current hidden state are combined into an attention vector . Lastly, a softmax layer predicts the next token in the output sequence: .
We add two auxiliary decoders when applying CVT. The auxiliary decoders share embedding and LSTM parameters with the primary decoder, but have different parameters for the attention mechanisms and softmax layers. For the first one, we restrict its view of the input by applying attention dropout, randomly zeroing out a fraction of its attention weights. The second one is trained to predict the next word in the target sequence rather than the current one: . Since there is no target sequence for unlabeled examples, we cannot apply teacher forcing to get an output distribution over the vocabulary from the primary decoder at each time step. Instead, we produce hard targets for the auxiliary modules by running the primary decoder with beam search on the input sequence. This idea has previously been applied to sequence-level knowledge distillation by Kim and Rush (2016) and makes the training procedure similar to back-translation (Sennrich et al., 2016).
We compare Cross-View Training against several strong baselines on seven tasks:
Combinatory Categorial Grammar (CCG) Supertagging: We use data from CCGBank (Hockenmaier and Steedman, 2007).
Text Chunking: We use the CoNLL-2000 data (Tjong Kim Sang and Buchholz, 2000).
Named Entity Recognition (NER): We use the CoNLL-2003 data (Tjong Kim Sang and De Meulder, 2003).
Fine-Grained NER (FGN): We use the OntoNotes (Hovy et al., 2006) dataset.
Part-of-Speech (POS) Tagging: We use the Wall Street Journal portion of the Penn Treebank (Marcus et al., 1993).
Dependency Parsing: We use the Penn Treebank converted to Stanford Dependencies version 3.3.0.
Machine Translation: We use the English-Vietnamese translation dataset from IWSLT 2015 (Cettolo et al., 2015).
We report (tokenized) BLEU scores on the tst2013 test set.
We use the 1 Billion Word Language Model Benchmark (Chelba et al., 2014) as a pool of unlabeled sentences for semi-supervised learning.
|Shortcut LSTM (Wu et al., 2017)||95.1||97.53|
|ID-CNN-CRF (Strubell et al., 2017)||90.7||86.8|
|JMT (Hashimoto et al., 2017)||95.8||97.55||94.7||92.9|
|TagLM* (Peters et al., 2017)||96.4||91.9|
|ELMo* (Peters et al., 2018)||92.2|
|Biaffine (Dozat and Manning, 2017)||95.7||94.1|
|Stack Pointer (Ma et al., 2018)||95.9||94.2|
|Stanford (Luong and Manning, 2015)||23.3|
|Google (Luong et al., 2017)||26.1|
|Virtual Adversarial Training*||95.1||95.1||91.8||87.9||97.64||95.4||93.7||–|
|ELMo (our implementation)*||95.8||96.5||92.2||88.5||97.72||96.2||94.4||29.3|
|ELMo + Multi-task*||95.9||96.8||92.3||88.4||97.79||96.4||94.8||–|
|CVT + Multi-task*||96.0||96.9||92.4||88.4||97.76||96.4||94.8||–|
|CVT + Multi-task + Large*||96.1||97.0||92.6||88.8||97.74||96.6||95.0||–|
4.1 Model Details and Baselines
We apply dropout during training, but not when running the primary prediction module to produce soft targets on unlabeled examples. In addition to the auxiliary prediction modules listed in Section 3, we find it slightly improves results to add another one that sees the whole input rather than a subset (but unlike the primary prediction module, does have dropout applied to its representations). Unless indicated otherwise, our models have LSTMs with 1024-sized hidden states and 512-sized projection layers. See the appendix for full training details and hyperparameters. We compare CVT with the following other semi-supervised learning algorithms:
Word Dropout. In this method, we only train the primary prediction module. When acting as a teacher it is run as normal, but when acting as a student, we randomly replace some of the input words with a REMOVED token. This is similar to CVT in that it exposes the model to a restricted view of the input. However, it is less data efficient. By carefully designing the auxiliary prediction modules, it is possible to train the auxiliary prediction modules to match the primary one across many different views of the input a once, rather than just one view at a time.
Virtual Adversarial Training (VAT). VAT (Miyato et al., 2016) works like word dropout, but adds noise to the word embeddings of the student instead of dropping out words. Notably, the noise is chosen adversarially so it most changes the model’s prediction. This method was applied successfully to semi-supervised text classification by Miyato et al. (2017a).
ELMo. ELMo incorporates the representations from a large separately-trained language model into a task-specific model. Our implementaiton follows Peters et al. (2018). When combining ELMo with multi-task learning, we allow each task to learn its own weights for the ELMo embeddings going into each prediction module. We found applying dropout to the ELMo embeddings was crucial for achieving good performance.
Of the prior results listed in Table 1, only TagLM and ELMo are semi-supervised. These methods first train an enormous language model on unlabeled data and incorporate the representations produced by the language model into a supervised classifier. Our base models use 1024 hidden units in their LSTMs (compared to 4096 in ELMo), require fewer training steps (around one pass over the billion-word benchmark rather than many passes), and do not require a pipelined training procedure. Therefore, although they perform on par with ELMo, they are faster and simpler to train. Increasing the size of our CVT+Multi-task model so it has 4096 units in its LSTMs like ELMo improves results further so they are significantly better than the ELMo+Multi-task ones. We suspect there could be further gains from combining our method with language model pre-training, which we leave for future work.
CVT + Multi-Task. We train a single shared-encoder CVT model to perform all of the tasks except machine translation (as it is quite different and requires more training time than the other ones). Multi-task learning improves results on all of the tasks except fine-grained NER, sometimes by large margins. Prior work on many-task NLP such as Hashimoto et al. (2017) uses complicated architectures and training algorithms. Our result shows that simple parameter sharing can be enough for effective many-task learning when the model is big and trained on a large amount of data.
Interestingly, multi-task learning works better in conjunction with CVT than with ELMo. We hypothesize that the ELMo models quickly fit to the data primarily using the ELMo vectors, which perhaps hinders the model from learning effective representations that transfer across tasks. We also believe CVT alleviates the danger of the model “forgetting” one task while training on the other ones, a well-known problem in many-task learning (Kirkpatrick et al., 2017). During multi-task CVT, the model makes predictions about unlabeled examples across all tasks, creating (artificial) all-tasks-labeled examples, so the model does not only see one task at a time. In fact, multi-task learning plus self training is similar to the Learning without Forgetting algorithm (Li and Hoiem, 2016), which trains the model to keep its predictions on an old task unchanged when learning a new task. To test the value of all-tasks-labeled examples, we trained a multi-task CVT model that only computes on one task at a time (chosen randomly for each unlabeled minibatch) instead of for all tasks in parallel. The one-at-a-time model performs substantially worse (see Table 2).
Model Generalization. In order to evaluate how our models generalize to the dev set from the train set, we plot the dev vs. train accuracy for our different methods as they learn (see Figure 4). Both CVT and multi-task learning improve model generalization: for the same train accuracy, the models get better dev accuracy than purely supervised learning. Interestingly, CVT continues to improve in dev set accuracy while close to 100% train accuracy for CCG, Chunking, and NER, perhaps because the model is still learning from unlabeled data even when it has completely fit to the train set. We also show results for a smaller multi-task + CVT model. Although it generalizes at least as well as the larger one, it halts making progress on the train set earlier. This suggests it is important to use sufficiently large neural networks for multi-task learning: otherwise the model does not have the capacity to fit to all the training data.
Auxiliary Prediction Module Ablation. We briefly explore which auxiliary prediction modules are more important for the sequence tagging tasks in Table 3. We find that both kinds of auxiliary prediction modules improve performance, but that the future and past modules improve results more than the forward and backward ones, perhaps because they see a more restricted and challenging view of the input.
Training Models on Small Datasets. We explore how CVT scales with dataset size by varying the amount of training data the model has access to. Unsurprisingly, the improvement of CVT over purely supervised learning grows larger as the amount of labeled data decreases (see Figure 5, left). Using only 25% of the labeled data, our approach already performs as well or better than a fully supervised model using 100% of the training data, demonstrating that CVT is particularly useful on small datasets.
Training Larger Models. Most sequence taggers and dependency parsers in prior work use small LSTMs (hidden state sizes of around 300) because larger models yield little to no gains in performance (Reimers and Gurevych, 2017). We found our own supervised approaches also do not benefit greatly from increasing the model size. In contrast, when using CVT accuracy scales better with model size (see Figure 5, right). This finding suggests the appropriate semi-supervised learning methods may enable the development of larger, more sophisticated models for NLP tasks with limited amounts of labeled data.
Generalizable Representations. Lastly, we explore training the CVT+multi-task model on five tasks, freezing the encoder, and then only training a prediction module on the sixth task. This tests whether the encoder’s representations generalize to a new task not seen during its training. Only training the prediction module is very fast because (1) the encoder (which is by far the slowest part of the model) has to be run over each example only once and (2) we do not back-propagate into the encoder. Results are shown in Table 4.
Training only a prediction module on top of multi-task representations works remarkably well, outperforming ELMo embeddings and sometimes even a vanilla supervised model, showing the multi-task model is building up effective representations for language. In particular, the representations could be used like skip-thought vectors (Kiros et al., 2015) to quickly train models on new tasks without slow representation learning.
5 Related Work
Unsupervised Representation Learning. Early approaches to deep semi-supervised learning pre-train neural models on unlabeled data, which has been successful for applications in computer vision (Jarrett et al., 2009; LeCun et al., 2010) and NLP. Particularly noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018). Pre-training on other tasks such as machine translation has also been studied (McCann et al., 2017). Other approaches train “thought vectors” representing sentences through unsupervised (Kiros et al., 2015; Hill et al., 2016) or supervised (Conneau et al., 2017) learning.
Self-Training. One of the earliest approaches to semi-supervised learning is self-training (Scudder, 1965), which has been successfully applied to NLP tasks such as word-sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006). In each round of training, the classifier, acting as a “teacher,” labels some of the unlabeled data and adds it to the training set. Then, acting as a “student,” it is retrained on the new training set. Many recent approaches (including the consistentency regularization methods discussed below and our own method) train the student with soft targets from the teacher’s output distribution rather than a hard label, making the procedure more akin to knowledge distillation (Hinton et al., 2015). It is also possible to use multiple models or prediction modules for the teacher, such as in tri-training (Zhou and Li, 2005; Ruder and Plank, 2018).
Consistency Regularization. Recent works add noise (e.g., drawn from a Gaussian distribution) or apply stochastic transformations (e.g., horizontally flipping an image) to the student’s inputs. This trains the model to give consistent predictions to nearby data points, encouraging distributional smoothness in the model. Consistency regularization has been very successful for computer vision applications (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017). However, stochastic input alterations are more difficult to apply to discrete data like text, making consistency regularization less used for natural language processing. One solution is to add noise to the model’s word embeddings (Miyato et al., 2017a); we compare against this approach in our experiments. CVT is easily applicable to text because it does not require changing the student’s inputs.
Multi-View Learning. Multi-view learning on data where features can be separated into distinct subsets has been well studied (Xu et al., 2013). Particularly relevant are co-training (Blum and Mitchell, 1998) and co-regularization (Sindhwani and Belkin, 2005), which trains two models with disjoint views of the input. On unlabeled data, each one acts as a “teacher” for the other model. In contrast to these methods, our approach trains a single unified model where auxiliary prediction modules see different, but not necessarily independent views of the input.
Self Supervision. Self-supervised learning methods train auxiliary prediction modules on tasks where performance can be measured without human-provided labels. Recent work has jointly trained image classifiers with tasks like relative position and colorization (Doersch and Zisserman, 2017), sequence taggers with language modeling (Rei, 2017), and reinforcement learning agents with predicting changes in the environment (Jaderberg et al., 2017). Unlike these approaches, our auxiliary losses are based on self-labeling, not labels deterministically constructed from the input.
Multi-Task Learning. There has been extensive prior work on multi-task learning (Caruana, 1997; Ruder, 2017). For NLP, most work has focused on a small number of closely related tasks (Luong et al., 2016; Zhang and Weiss, 2016; Søgaard and Goldberg, 2016; Peng et al., 2017). Many-task systems are less commonly developed. Collobert and Weston (2008) propose a many-task system sharing word embeddings between the tasks, Hashimoto et al. (2017) train a many-task model where the tasks are arranged hierarchically according to their linguistic level, and Subramanian et al. (2018) train a shared-encoder many-task model for the purpose of learning better sentence representations for use in downstream tasks, not for improving results on the original tasks.
We propose Cross-View Training, a new method for semi-supervised learning. Our approach allows models to effectively leverage their own predictions on unlabeled data, training them to produce effective representations that yield accurate predictions even when some of the input is not available. We achieve excellent results across seven NLP tasks, especially when CVT is combined with multi-task learning.
We thank Abi See, Christopher Clark, He He, Peng Qi, Reid Pryzant, Yuaho Zhang, and the anonymous reviewers for their thoughtful comments and suggestions. We thank Takeru Miyato for help with his virtual adversarial training code and Emma Strubell for answering our questions about OntoNotes NER. Kevin is supported by a Google PhD Fellowship.
- Bachman et al. (2014) Philip Bachman, Ouais Alsharif, and Doina Precup. 2014. Learning with pseudo-ensembles. In NIPS.
- Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
- Blum and Mitchell (1998) Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In COLT. ACM.
- Caruana (1997) Rich Caruana. 1997. Multitask learning. Machine Learning, 28:41–75.
- Cettolo et al. (2015) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. The IWSLT 2015 evaluation campaign. In International Workshop on Spoken Language Translation.
- Chelba et al. (2014) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH.
- Chiu and Nichols (2016) Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics.
- Choe and Charniak (2016) Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In EMNLP.
- Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML.
- Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research.
- Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP.
- Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS.
- Dai et al. (2017) Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan Salakhutdinov. 2017. Good semi-supervised learning that requires a bad gan. In NIPS.
- Doersch and Zisserman (2017) Carl Doersch and Andrew Zisserman. 2017. Multi-task self-supervised visual learning. arXiv preprint arXiv:1708.07860.
- Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR.
- Furlanello et al. (2018) Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In ICML.
- Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5):602–610.
- Hashimoto et al. (2017) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In EMNLP.
- Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In HLT-NAACL.
- Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
- Hinton et al. (2012) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
- Hockenmaier and Steedman (2007) Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn treebank. Computational Linguistics, 33(3):355–396.
- Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In HLT-NAACL.
- Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL.
- Jaderberg et al. (2017) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2017. Reinforcement learning with unsupervised auxiliary tasks. In ICLR.
- Jarrett et al. (2009) Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. 2009. What is the best multi-stage architecture for object recognition? In IEEE Conference on Computer Vision.
- Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP.
- Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114 13:3521–3526.
- Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302.
- Krizhnevsky and Hinton (2009) Alex Krizhnevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images.
- Kuncoro et al. (2017) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In EACL.
- Laine and Aila (2017) Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In ICLR.
- Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In ACL.
- LeCun et al. (2010) Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. 2010. Convolutional networks and applications in vision. In ISCAS. IEEE.
- Lewis et al. (2016) Mike Lewis, Kenton Lee, and Luke Zettlemoyer. 2016. LSTM CCG parsing. In HLT-NAACL.
- Li and Hoiem (2016) Zhizhong Li and Derek Hoiem. 2016. Learning without forgetting. In ECCV.
- Liu and Zhang (2017) Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. TACL.
- Liu et al. (2017) Liyuan Liu, Jingbo Shang, Frank Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2017. Empower sequence labeling with task-aware neural language model. arXiv preprint arXiv:1709.04109.
- Luong et al. (2017) Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. 2017. Neural machine translation (seq2seq) tutorial. https://github.com/tensorflow/nmt.
- Luong et al. (2016) Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR.
- Luong and Manning (2015) Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domains. In IWSLT.
- Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.
- Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNN-CRF. In ACL.
- Ma and Hovy (2017) Xuezhe Ma and Eduard Hovy. 2017. Neural probabilistic model for non-projective mst parsing. In IJCNLP.
- Ma et al. (2018) Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In ACL.
- Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The Penn treebank. Computational linguistics, 19(2):313–330.
- McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS.
- McClosky et al. (2006) David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In ACL.
- Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
- Miyato et al. (2017a) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017a. Adversarial training methods for semi-supervised text classification. In ICLR.
- Miyato et al. (2017b) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2017b. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. arXiv preprint arXiv:1704.03976.
- Miyato et al. (2016) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In ICLR.
- Park et al. (2017) Sungrae Park, Jun-Keon Park, Su-Jin Shin, and Il-Chul Moon. 2017. Adversarial dropout for supervised and semi-supervised learning. arXiv preprint arXiv:1707.03631.
- Peng et al. (2017) Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In ACL.
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
- Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In ICLR.
- Peters et al. (2017) Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL.
- Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
- Polyak (1964) Boris T Polyak. 1964. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17.
- Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised.
- Ramachandran et al. (2017) Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In EMNLP.
- Rei (2017) Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In ACL.
- Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In EMNLP.
- Ruder (2017) Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
- Ruder and Plank (2018) Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In ACL.
- Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NIPS.
- Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In NIPS.
- Scudder (1965) H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371.
- Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL.
- Sindhwani and Belkin (2005) Vikas Sindhwani and Mikhail Belkin. 2005. A co-regularization approach to semi-supervised learning with multiple views. In ICML Workshop on Learning with Multiple Views.
- Søgaard and Goldberg (2016) Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL.
- Strubell et al. (2017) Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate sequence labeling with iterated dilated convolutions. In EMNLP.
- Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In ICLR.
- Sutskever et al. (2013) Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In ICML.
- Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS.
- Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR.
- Tarvainen and Valpola (2017) Antti Tarvainen and Harri Valpola. 2017. Weight-averaged consistency targets improve semi-supervised deep learning results. In Workshop on Learning with Limited Labeled Data, NIPS.
- Tjong Kim Sang and Buchholz (2000) Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In CoNLL.
- Tjong Kim Sang and De Meulder (2003) Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In HLT-NAACL.
- Verma et al. (2018) Vikas Verma, Alex Lamb, Christopher Beckham, Aaron Courville, Ioannis Mitliagkis, and Yoshua Bengio. 2018. Manifold mixup: Encouraging meaningful on-manifold interpolation as a regularizer. arXiv preprint arXiv:1806.05236.
- Wei et al. (2018) Xiang Wei, Zixia Liu, Liqiang Wang, and Boqing Gong. 2018. Improving the improved training of Wasserstein GANs. In ICLR.
- Wu et al. (2017) Huijia Wu, Jiajun Zhang, and Chengqing Zong. 2017. Shortcut sequence tagging. arXiv preprint arXiv:1701.00576.
- Xu et al. (2013) Chang Xu, Dacheng Tao, and Chao Xu. 2013. A survey on multi-view learning. arXiv preprint arXiv:1304.5634.
- Yarowsky (1995) David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In ACL.
- Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In ICLR.
- Zhang and Weiss (2016) Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In ACL.
- Zhou and Li (2005) Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering.
Appendix A Detailed Results
We provide a more detailed version of the test set results in the paper, adding two decimals of precision, standard deviations of the 5 runs for each model, and more prior work, in Table 5.
Appendix B Model Details
Our models use two layer CNN-BiLSTM encoders (Chiu and Nichols, 2016; Ma and Hovy, 2016; Lample et al., 2016) and task-specific prediction modules. See Section 3 of the paper for details. We provide a few minor details not covered there below.
Sequence Tagging. For Chunking and Named Entity Recognition, we use a BIOES tagging scheme. We apply label smoothing (Szegedy et al., 2016; Pereyra et al., 2017) with a rate of 0.1 to the target labels when training on the labeled data.
Dependency Parsing. We omit punctuation from evaluation, which is standard practice for the PTB-SD 3.3.0 dataset. ROOT is represented with a fixed vector instead of using a vector from the encoder, but otherwise dependencies coming from ROOT are scored the same way as the other dependencies.
Machine Translation. We apply dropout to the output of each LSTM layer in the decoder. Our implementation is heavily based off of the Google NMT Tutorial333https://github.com/tensorflow/nmt (Luong et al., 2017). We attribute our significantly better results to using pre-trained word embeddings, a character-level CNN, a larger model, stronger regularization, and better hyperparameter tuning. Target words occurring 5 or fewer times in the train set are replaced with a UNK token (but not during evaluation). We use a beam size of 10 when performing beam search. We found it slightly beneficial to apply label smoothing with a rate of 0.1 to the teacher’s predictions (unlike our other tasks, the teacher only provides hard targets to the students for translation).
Multi-Task Learning. Several of our datasets are constructed from the Penn Treebank. However, we treat them as separate rather than providing examples labeled across multiple tasks to our model during supervised training. Furthermore, the Penn Treebank tasks do not all use the same train/dev/test splits. We ensure the training split of one task never overlaps the evaluation split of another by discarding the overlapping examples from the train sets.
Other Details. We apply dropout (Hinton et al., 2012) to the word embeddings and outputs of each Bi-LSTM. We use an exponential-moving-average (EMA) of the model weights from training for the final model; we found this to slightly improve accuracy and significantly reduce the variance in accuracy between models trained with different random initializations. The model is trained using SGD with momentum (Polyak, 1964; Sutskever et al., 2013). Word embeddings are initialized with GloVe vectors (Pennington et al., 2014) and fine-tuned during training. The full set of model hyperparameters are listed in Table 6.
Baselines. Baselines were run with the same architecture and hyperparameters as the CVT model. For the “word dropout” model, we randomly replace words in the input sentence with a REMOVED token with probability 0.1 (this value worked well on the dev sets). For Virtual Adversarial Training, we set the norm of the perturbation to be 1.5 for CCG, 1.0 for Dependency Parsing, and 0.5 for the other tasks (these values worked best on the dev sets). Otherwise, the implementation is as described in (Miyato et al., 2017a); we based our implementation off of their code444https://github.com/tensorflow/models/tree/master/research/adversarial_text. We were unable to successfully apply VAT to machine translation, perhaps because the student is provided hard targets for that task. For ELMo, we applied dropout to the ELMo embeddings before they are incorporated into the rest of the model. When training the multi-task ELMo model, each prediction module has its own set of softmax-normalized weights ( in (Peters et al., 2018)) for the ELMo emeddings going into the task-specific prediction modules. All tasks share the same weights for the ELMo embeddings going into the shared Bi-LSTM encoder.
|LSTM-CNN-CRF (Ma and Hovy, 2016)||91.21||97.55|
|LSTM-CNN (Chiu and Nichols, 2016)||91.62 0.33||86.28 0.26|
|ID-CNN-CRF (Strubell et al., 2017)||90.65 0.15||86.84 0.19|
|Tri-Trained LSTM (Lewis et al., 2016)||94.7|
|Shortcut LSTM (Wu et al., 2017)||95.08||97.53|
|JMT* (Hashimoto et al., 2017)||95.77||97.55||94.67||92.90|
|LM-LSTM-CNN-CRF (Liu et al., 2017)||95.96 0.08||91.71 0.10||97.53 0.03|
|TagLM (Peters et al., 2017)||96.37 0.05||91.93 0.19|
|ELMo (Peters et al., 2018)||92.22 0.10|
|NPM (Ma and Hovy, 2017)||94.9||93.0|
|Deep Biaffine (Dozat and Manning, 2017)||95.74||94.08|
|Stack Pointer (Ma et al., 2018)||95.87||94.19|
|Stanford (Luong and Manning, 2015)||23.3|
|Google (Luong et al., 2017)||26.1|
|Supervised||94.94 0.02||95.10 0.06||91.16 0.09||87.48 0.08||97.60 0.02||95.08 0.03||93.27 0.03||28.88 0.12|
|Virtual Adversarial Training*||95.07 0.04||95.06 0.06||91.75 0.10||87.91 0.11||97.64 0.03||95.44 0.06||93.72 0.07||–|
|Word Dropout*||95.20 0.04||95.79 0.08||92.14 0.11||88.06 0.09||97.66 0.01||95.56 0.05||93.80 0.08||29.33 0.10|
|ELMo*||95.79 0.04||96.50 0.03||92.24 0.09||88.49 0.12||97.72 0.01||96.22 0.05||94.44 0.06||29.34 0.11|
|ELMo + Multi-task*||95.91 0.05||96.83 0.03||92.32 0.12||88.37 0.16||97.79 0.03||96.40 0.04||94.79 0.05||–|
|CVT*||95.65 0.04||96.58 0.04||92.34 0.06||88.68 0.14||97.70 0.03||95.86 0.03||94.06 0.02||29.58 0.07|
|CVT + Multi-Task*||95.97 0.04||96.85 0.05||92.42 0.08||88.42 0.13||97.76 0.02||96.44 0.04||94.83 0.06||–|
|CVT + Multi-Task + Large*||96.05 0.03||96.98 0.05||92.61 0.09||88.81 0.09||97.74 0.02||96.61 0.04||95.02 0.04||–|
|Word Embeddings Initializiation||300d GloVe 6B|
|Character Embedding Size||50|
|Character CNN Filter Widths||[2, 3, 4]|
|Character CNN Num Filters||300 (100 per filter width)|
|Encoder LSTM sizes||1024 for the first layer, 512 for the second one|
|Encoder LSTM sizes, “Large” model||4096 for the first layer, 2048 for the second one|
|LSTM projection layer size||512|
|Hidden layer sizes||512|
|Dropout||0.5 for labeled examples, 0.8 for unlabeled examples|
|Learning rate||( is number of SGD updates so far)|
|Batch size||64 sentences|
Appendix C CVT for Image Recognition
Although the focus of our work is on NLP, we also applied CVT to image recognition and found it performs competitively with existing methods. Most of the semi-supervised image recognition approaches we compare against rely on the inputs being continuous, so they would be difficult to apply to text. More specifically, consistency regularization methods (Sajjadi et al., 2016; Laine and Aila, 2017; Miyato et al., 2017b) rely on adding continuous noise and applying image-specific transformations like cropping to inputs, GANs (Salimans et al., 2016; Wei et al., 2018) are very difficult to train on text due to its discrete nature, and mixup (Zhang et al., 2018; Verma et al., 2018) requires a way of smoothly interpolating between different inputs.
Approach. Our image recognition models are based on Convolutional Neural Networks, which produce a set of features from an image . The first two dimensions of index into the spatial coordinates of feature vectors and is the size of the feature vectors. For shallower CNNs, a particular feature vector corresponds to a region of the input image. For example, would be a -dimensional vector of features extracted from the upper left corner. For deeper CNNs, a particular feature vector would be extracted from the whole image, but still only use a “region” of the representations from an earlier layer. The CNNs in our experiments are all in the first category.
The primary prediction layers of our CNNs take as input the mean of over the first two dimensions, which results in a -dimensional vector that is fed into a softmax layer:
We add auxiliary prediction layers to the top of the CNN. The th layer takes a single feature vector as input:
Data. We evaluated our models on the CIFAR-10 (Krizhnevsky and Hinton, 2009) dataset. Following previous work, we make the datasets semi-supervised by only using the provided labels for a subset of the examples in the training set; the rest are treated as unlabeled examples.
Model. We use the convolutional neural network from Miyato et al. (2017b), adapting their TensorFlow implementation555https://github.com/takerum/vat_tf. Their model contains 9 convolutional layers and 2 max pooling layers. See Appendix D of Miyato et al.’s paper for more details.
We add 36 auxiliary softmax layers to the collection of feature vectors produced by the CNN. Each auxiliary layer sees a patch of the image ranging in size from pixels (the corner) to pixels (the center) of the pixel images. For some experiments, we combine CVT with standard consistency regularization by adding a perturbation (e.g., a small random vector) to the student’s inputs when computing .
|GAN (Salimans et al., 2016)||–||18.63 2.32|
|Stochastic Transformations (Sajjadi et al., 2016)||–||11.29 0.24|
|model (Laine and Aila, 2017)||16.55 0.29||12.36 0.31|
|Temporal Ensemble (Laine and Aila, 2017)||–||12.16 0.24|
|Mean Teacher (Tarvainen and Valpola, 2017)||–||12.31 0.28|
|Complement GAN (Dai et al., 2017)||14.41 0.30||–|
|VAT (Miyato et al., 2017b)||13.15||10.55|
|VAdD (Park et al., 2017)||–||11.68 0.19|
|VAdD + VAT (Park et al., 2017)||–||10.07 0.11|
|SNGT + model (Luong et al., 2017)||13.62 0.17||11.00 0.36|
|SNGT + VAT (Luong et al., 2017)||12.49 0.36||9.89 0.34|
|Consistency + WGAN (Wei et al., 2018)||–||9.98 0.21|
|Manifold Mixup (Verma et al., 2018)||–||10.26 0.32|
|Supervised||23.61 0.60||19.61 0.56|
|VAT (ours)||13.29 0.33||10.90 0.31|
|CVT, no input perturbation||14.63 0.20||12.44 0.27|
|CVT, random input perturbation||13.80 0.30||11.10 0.26|
|CVT, adversarial input perturbation||12.01 0.11||10.11 0.15|
Results. The results are shown in Table 7. Unsurprisingly, adding continuous noise to the inputs works much better with images, where the inputs are naturally continuous, than with language. Therefore we see much better results from VAT on semi-supervised CIFAR-10 compared to on our NLP tasks. However, we still find incorporating CVT improves over models without CVT. Our CVT + VAT models are competitive with current start-of-the-art approaches. We found the gains from CVT are larger when no data augmentation is applied, perhaps because random translations of the input expose the model to different “views” in a similar manner as with CVT.
Appendix D Negative Results
We briefly describe a few ideas we implemented that did not seem to be effective in initial experiments. Note these findings are from early one-off experiments. We did not pursue them further after our first attempts did not pan out, so it is possible that some of these approaches could be effective with the proper adjustments and tuning.
Hard vs soft targets: Classic self-training algorithms train the student model with one-hot “hard” targets corresponding to the teacher’s highest probability prediction. In our experiments, this decreased performance compared to using soft targets. This finding is consistent with research on knowledge distillation (Hinton et al., 2015; Furlanello et al., 2018) where soft targets also work notably better than hard targets.
Confidence thresholding: Classic self-training often only trains the student on a subset of the unlabeled examples on which the teacher has confident predictions (i.e., the output distribution has low entropy). We tried both “hard” (where the student ignores low-confidence examples) and “soft” (where examples are weighted according to the teacher’s confidence) versions of this for training our models, but they did not seem to improve performance.
Mean Teacher: The Mean Teacher method (Tarvainen and Valpola, 2017) tracks an exponential moving average (EMA) of model weights, which are used to produce targets for the students. The idea is that these targets may be better quality due to a self-ensembling effect. However, we found this approach to have little to no benefit in our experiments, although using EMA model weights at test time did improve results slightly.
Purely supervised CVT: Lastly, we explored adding cross-view losses to purely supervised classifiers. We hoped that adding auxiliary softmax layers with different views of the input would act as a regularizer on the model. However, we found little to no benefit from this approach. This negative result suggests that the gains from CVT are from the improved semi-supervised learning mechanism, not the additional prediction layers regularizing the model.