Towards Open Intent Discovery for Conversational Text
Detecting and identifying user intent from text, both written and spoken, plays an important role in modelling and understand dialogs. Existing research for intent discovery model it as a classification task with a predefined set of known categories. To generailze beyond these preexisting classes, we define a new task of open intent discovery. We investigate how intent can be generalized to those not seen during training. To this end, we propose a two-stage approach to this task - predicting whether an utterance contains an intent, and then tagging the intent in the input utterance. Our model consists of a bidirectional LSTM with a CRF on top to capture contextual semantics, subject to some constraints. Self-attention is used to learn long distance dependencies. Further, we adapt an adversarial training approach to improve robustness and perforamce across domains. We also present a dataset of 25k real-life utterances that have been labelled via crowd sourcing. Our experiments across different domains and real-world datasets show the effectiveness of our approach, with less than 100 annotated examples needed per unique domain to recognize diverse intents. The approach outperforms state-of-the-art baselines by 5-15% F1 score points.
Recent advances in the efficacy and accuracy of natural language understanding (NLU) and speech recognition technologies have triggered the advent of a wealth of conversational agents such as Apple’s Siri, Microsoft’s Cortana and Amazon’s Alexa. In order to effectively and intelligently interact with people and answer their diverse questions, such agents need to parse and interpret human language utterances, especially people’s intentions, and respond accordingly. The problem of recognizing human intentions or intents from their text or speech inputs has several downstream applications. It can help in summarizing the common or frequent user objectives and functions associated with a business or a product. It can highlight and help prioritize common bugs and issues reported to customer support or public forums, and spot action items in emails or meeting transcripts. Progress in the field of deep learning has led to the emergence of models that can detect user intents (Chen et al., 2013; Bhargava et al., 2013; Xu and Sarikaya, 2013; Gupta et al., 2014; Wang et al., 2015; Kim et al., 2016; Liu and Lane, 2016; Zhang and Wang, 2016; Kim et al., 2017a; Goo et al., 2018; Coucke et al., 2018; Xia et al., 2018) and identify semantically relevant entities linked with those intents (slot filling).
Most existing research including commercial NLU engines formulate the problem of identifying user intents as a multi-class classification task. Assuming the presence of an intent in a given user utterance, such models categorize the utterance into pre-defined intent classes for which sufficient labeled data is available during the model training phase. Most techniques are unable to address inputs that belong to new or previously unseen intent categories, i.e., they work with a closed world assumption. Studies further assume that an input text expresses only a single intent (e.g. (Liu and Lane, 2016; Kim et al., 2016; Goo et al., 2018; Coucke et al., 2018; Xia et al., 2018)). This is very much unlike real-world scenarios where users often express multiple, distinct intentions within one dialog turn or utterance.
In our work, we propose a framework called TOP-ID (Towards OPen Intent Discovery). It automatically discovers user intents in natural language without prior knowledge of a comprehensive list of intent classes that the text may comprise of. In other words, TOP-ID is not restricted to a pre-defined set of intent categories. It can recognize instances of intent types that it has never seen before. This is a much more challenging problem than the multi-class classification problem that prior literature generally formulates the intent detection task as. We therefore name this novel task as Open Intent Discovery. It focuses on identifying and extracting user intentions from text utterances explicitly containing them in their content. It does not infer or deduce the intent if it is implicitly stated in the text. To illustrate, the text “Please make a 10:30 sharp appointment for a haircut" contains a single intent of making a haircut appointment; whereas the text “I would like to reserve a seat and also if possible, request a special meal on my flight" contains multiple intents – a seat reservation and a meal request. Contrarily, the sentence “Anyone knows the battery life of iPhone?" merely requests information on a particular topic and does not seem to contain any tangible intent action, such as that of buying an iPhone. We do not consider such ambiguous or questionable utterances.
Recent work by Xia et al (Xia et al., 2018) has a similar objective as ours, i.e., recognizing intents outside of the labeled training data. They treat this as a zero-shot classification problem. However, their method can only handle the basic case of an input utterance containing a single intent. It also requires a list of new or unseen intent classes to be available at test time, and classifies the text input into one of them. Our work does not have these restrictions, and to the best of our knowledge, is the first work to address the aforementioned limitations. It gives a fine-grained picture of the diverse user intents in an utterance or a collection of utterances, rather than merely grouping intents into higher level categories.
Our proposed two-stage approach, TOP-ID, aims to solve the problem of open intent discovery. In the first stage, our method employs a softmax classifier on top of a bidirectional LSTM to determine if the text input is likely to contain an explicit, tangible intent or not. If it does, the second stage of TOP-ID is applied to identify and extract all possible intents in a consistent and generalizable format. We model this as a sequence tagging problem. We solve it by developing a neural network model consisting of a Conditional Random Field (CRF) on top of a bidirectional LSTM, accompanied by a multi-head self-attention mechanism. A crucial challenge associated with developing a generic technique for open intent discovery is ensuring its effectiveness it across several task domains or fields. TOP-ID represents all kinds of user intents extracted from the textual input in a common format, independent of their domain. We further employ adversarial training at the lower layers of our model, and pre-train it without supervision in the target domain under consideration. These strategies empower our model for cross-domain adaptation even in the absence of sufficient labeled training data, as we show empirically in Section 6. Moreover, commonly used intent-labeled datasets in dialog research such as SNIPS (Coucke et al., 2018) or ATIS (Hemphill et al., 1990; Dahl et al., 1994) largely have concise, coherent and single-sentence texts. They are not very representative of complex, real-world dialog scenarios which could be verbose and ungrammatical, with intents scattered throughout their content. Therefore, we develop a large dataset with 25K real-world utterances from the online question-answer forum of Stack Exchange. They span several genres and have been curated for intents by crowd workers.
To summarize, the key contributions of our work are:
We formulate and solve the novel problem of open intent discovery in text utterances. Our proposed two-stage technique TOP-ID is flexible, generalizable, and agnostic of the domain of the target text.
TOP-ID, can discover both previously seen as well as unseen (during training) user intents in diverse real-world scenarios. It can identify multiple user intents per utterance.
We curate and present a large, intent-annotated dataset of 25K text instances from real-world task domains, without any restriction on the number or types of intents possible.
2. Related Work
Prior work on user intents encompasses two avenues: asynchronous, written communication (forums, blogs, tweets) and synchronous dialog. In both cases, intent detection is frequently modeled as a binary or multi-class classification problem, with each class representing the presence or absence of a specific kind of intent. Supervised and semi-supervised learning models based on linguistic and sentiment features have been used to model racial intent on Tumblr (Agarwal and Sureka, 2017), and purchase intent in social posts (Gupta et al., 2014; Wang et al., 2015) and in discussion forums (Chen et al., 2013). Modeling responses for dialog agents such as Microsoft LUIS111https://www.luis.ai/, Google Dialogflow222https://dialogflow.com/, and Amazon Lex333https://aws.amazon.com/lex/ includes the dual tasks of domain-aware intent detection and slot filling. Recent approaches have drawn upon progress in CNN, RNN, and CRF based language models to improve intent detection (Xu and Sarikaya, 2013; Mesnil et al., 2015; Kim et al., 2016; Liu and Lane, 2016; Zhang and Wang, 2016; Kim et al., 2017a; Goo et al., 2018). Performing slot filling jointly with intent detection has improved the performance of both tasks (Kim et al., 2016; Liu and Lane, 2016; Zhang and Wang, 2016).
In the health community, Cai et al (Cai et al., 2017) used hierarchical clustering to learn a taxonomy of intent classes, and applied a hybrid CNN-LSTM model to classify the intent of medical queries. We take this idea further and learn to identify arbitrary intents beyond even a predefined taxonomy. Improvements in intent detection and slot filling based on adversarial learning have also been explored (Liu and Lane, 2017; Kim et al., 2017b; Yu and Lam, 2018). We exploit adversarial training to generate adversarial input examples for improving the performance of our model and for domain adaptation (Section 5.1). Moreover, adding linguistic structure to existing models has been shown to improve their performance across a wide range of NLU tasks, such as word embedding (Mrkšić et al., 2016), machine translation (Chen et al., 2017), named entity recognition (Jochim and Deleris, 2017), and semantic role labelling (He et al., 2017). We impose linguistic constraints on the CRF layer of TOP-ID to preserve the semantics of intent actions and their associated objects (Section 5.3.1).
3. Problem Formulation
This work introduces and addresses the novel problem of Open Intent Discovery in written, asynchronous text conversations. The objective of this problem is to accurately determine the existence of one or more actionable intents from the text input, and subsequently identify and extract all the possible user intents. These intents may be underlying goals, activities or tasks that a user wants to perform or have performed. We therefore define an intent as a text phrase consisting of two parts: (i) an action, which is a word or phrase representing a tangible purpose, task or activity which is to be requested or performed, and (ii) an object, which represents those entity words or phrases that the action is going to act or operate upon. A similar definition has been used in prior literature to define intention posts in social media and discussion forums (Wang et al., 2015; Chen et al., 2013). For instance, the intent of the author of the text “Please make a 10:30 sharp appointment for a haircut" is to make or schedule a haircut appointment. The intent consists of an action “make" and an object “appointment", “appointment for haircut", or “haircut appointment". Similarly, consider the utterance “I would like to reserve a seat and request a special meal on my flight". In this case, the actions are “reserve" and “request" and the objects are seat and special meal, for the respective intents of seat booking and meal request.
Following our two-part definition of an intent, we formulate the problem of open intent discovery as a sequence tagging problem. We consider three tags: Action, Object, and None (denotes all the remaining words or phrases in the text utterance that are neither an Action nor an Object). A user intent then consists of a matching pair of an Action phrase and an Object phrase. Employing such a consistent, generic representation enables our TOP-ID framework to identify and extract all possible forms of user intentions which fit in this format. These include previously unseen intent types that were not encountered while training, unlike a classifier that can only address a pre-defined set of intent categories. This can also help us discover multiple possible user intents from a single text inputs and not just a single intent, unlike most of the current literature. This is crucial since user queries or utterances can often consist of more than one intent in them such as a single main or principal intent, and a few more interlinked, accompanying intents. There could also be multiple tasks that may need to be accomplished together (e.g. reserve seat and request special meal). Besides, having a common format to represent an intent contributes immensely in finding user intents irrespective of their target domain or topic.
We observe from the previous illustrations that the Action component of an intent is likely to consist of a verb or infinitive phrase that follows a noun or a subject phrase. Further, the Object component of an intent often comprises of a noun or compound noun (i.e., an expression with multiple nouns) phrase, possibly qualified by adverbs or adjectives. Nevertheless, we cannot simply use a part-of-speech (POS) tagger or a language dependency parser for the purpose of identifying intents due to the following reasons. First, a POS tagger or a parser cannot distinguish between the various Action-Object pairs present. They will identify all pairs, including irrelevant ones that are merely part of the descriptive text and are not associated with user intent. They will hence suffer from a low precision problem, as we empirically show in Table 3. Second, corresponding Action and Object phrases may be spatially distant from each other in the input text and may even span multiple sentences (see Table 5). Having said that, we do notice the efficacy of initially pre-training the weights of the model of the second stage of TOP-ID with the verb-object tags obtained from a dependency parser, as we show empirically in Table 3. It helps our model learn generic indicators for various kinds of intents, independent of the topic or domain of the input text. We then fine-tune our model with labeled data specific to our problem. Pre-training can also help overcome the issue of overfitting in sequence learning models, especially if there is insufficient annotated training data.
TOP-ID (Figure 1) works in two stages for input utterances:
Open Intent Existence Recognition: The first stage uses a two-layered Bi-LSTM to semantically encode the input utterances. It then employs a binary sigmoid classifier to predict whether an actionable user intent exists within the text input. If not, our method ends here. Otherwise, it moves to the second stage of extracting the intents from the text.
Open Intent Extraction: The second stage builds upon state-of-the-art sequence tagging systems that utilize bidirectional RNNs and CRFs (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016). It uses a CRF on top of a Bi-LSTM accompanied by adversarial training and attention, to extract actionable user intents from the text input.
4. TOP-ID Stage I: Open Intent Existence
Given an input text consisting of a sequence of words , we first transform it into a feature sequence by constructing the character level representation of each word . The reason for this is that incorporating character level representations of words via convolutional neural networks (CNNs) can boost the effectiveness of sentence representations, by capturing morphological information present in the language (Zhang et al., 2015; Ma and Hovy, 2016). For this purpose, we build a CNN consisting of convolutional and max pooling layers with dropout (Srivastava et al., 2014), similar to Huang et al (Huang et al., 2015). We also obtain word level GloVe embeddings (Pennington et al., 2014) for each token from a pre-trained model that has been trained on Common Crawl, a giant corpus of web crawled data. Such low-dimensional and dense embeddings are highly effective in capturing both syntactic and semantic information. Nevertheless, character-level information can often be overshadowed by word-level embeddings if both are simply concatenated to produce a combined representation for each word. Hence, we adopt a highway network (Srivastava et al., 2015) to retain the impact of the both kinds of embeddings, and merge them in a balanced manner.
Let be the concatenation of the character and word level representations and of the word . The combined embedding from the highway network is then given by:
where is the hyperbolic tangent function, denotes the element-wise multiplication operation, and are weight matrices, and and are bias vectors. (transform gate) and (carry gate) are non-linear transformation functions indicating the proportion of output produced by transforming the input and carrying it. Every word is thus transformed into a -dimensional embedding . This embedding layer forms the first layer of our model architecture, and serves as input for the next layer, namely a Bi-LSTM layer.
Bidirectional LSTM (Graves et al., 2013) based models have been used for various sequence modeling domains where it is often beneficial to utilize both the past and future context. They are effective in encoding sentences by considering the order of their constituent words and can account for long-range word dependencies. Figure 2 displays the architecture of the first stage of TOP-ID. The Bi-LSTM layer contains two sub-layers for the forward () and backward () sequence contexts respectively. and are both generated based on the recurrences of an LSTM cell (Hochreiter and Schmidhuber, 1997), and their concatenation gives the combined output at time step .We use a stack of two Bi-LSTM layers to generate a sequence of word-level representations . To obtain a fixed length vector from the Bi-LSTM output, we perform one-dimensional max pooling over all ’s, followed by a sigmoid output layer. It performs a binary prediction of whether an actionable intent is present in the text input or not. We use 400 hidden LSTM units with L2 regularization. We apply dropout to the Bi-LSTM and max pooling layers with a probability value of 0.5, to avoid overfitting and co-adaptation of the hidden units. We train the model via the Adam (Kingma and Ba, 2014) optimizer with gradient clipping to optimize the binary cross entropy loss function. We set the initial learning rate to 0.001 with a decay of 0.05. If the first stage of our TOP-ID framework predicts the input utterance as containing an intent, it moves to the second stage, i.e. extracting the actual actionable intents from the input utterance.
5. TOP-ID Stage II: Open Intent Extraction
This is the second stage of our TOP-ID intent discovery framework. The initial layers of this intent extraction model (Figure 3), namely the input and embedding layers are constructed similar to stage I (Section 4). The Bi-LSTM structure on top of the embedding layer is also the same, except for the number of its layers (two Bi-LSTM layers for stage I, and one Bi-LSTM layer for stage II). The remaining components of the open intent extraction model are detailed below.
5.1. Adversarial Training
Adversarial training (Goodfellow et al., 2015; Miyato et al., 2016) is a useful technique to regularize predictive models and improve their robustness to small input perturbations. It has also been applied in the literature for the purpose of domain adaptation, i.e., to discover features and structures that are common across multiple domains (Ganin and Lempitsky, 2014; Kim et al., 2017b; Liu and Lane, 2017). Since both the above are goals of this work, we utilize adversarial training to enhance our model. We generate adversarial input examples that are very close to the original inputs and should yield the same labels, yet are likely to be mispredicted by the current model. These examples are created by adding small worst case perturbations or noise to the inputs in the direction that significantly increases the model’s loss function. Our model is then trained on the mixture of original and adversarial examples to improve its stability to input perturbations. Since adversarial training considers continuous perturbations to inputs, we add adversarial noise at the embedding layer, as in (Miyato et al., 2016).
Let an input text be represented by an embedding , as detailed in Section 4. We generate its worst case perturbation of a small bounded norm , which is a hyperparameter to be tuned. It maximizes the loss function of the current model with parameters as follows:
Since the exact computation of is intractable in complex neural networks, we use the first order approximation via the fast gradient method (Goodfellow et al., 2015; Miyato et al., 2016) to obtain an approximate worst case perturbation of norm . We also normalize the word and character embeddings, so that the model does not trivially learn the embeddings of large norms and make the perturbations insignificant (Miyato et al., 2016).
where represents an adversarial example generated from embedding . and represent the loss functions from the original training instance and its adversarial transformation respectively. is a weighting parameter. The new loss function can be optimized in the same way as the original loss .
5.2. Attention Mechanism
We employ attention to select and focus on the important and essential hidden states of the Bi-LSTM layer. In particular, we use a multi-head self-attention mechanism (Vaswani et al., 2017; Im and Cho, 2017; Tan et al., 2017; Lin et al., 2017) that jointly attends to information at different positions of the input sequence, with multiple individual attention functions and separately normalized parameters called attention heads. This enables it to capture different contexts in a fine-grained manner and learn long-range dependencies effectively. Each attention head computes a sequence from the output of the Bi-LSTM layer by projecting it to a key , a value , and a query via distinct affine transformations with ReLU activations (Glorot et al., 2011). The attention weights for attention head between word tokens and are computed as:
Here denotes an element-wise product and softmax indicates the softmax function along the -th dimension. The individual attention head outputs are concatenated into for token . The scaled dot product above enhances the optimization process by better distributing the gradients and flattening the softmax function (Vaswani et al., 2017).
5.3. Sequence Tagging via CRFs
The output of the attention layer serves as input to the next layer of our TOP-ID intent extraction model, namely a CRF (Lafferty et al., 2001). CRFs effectively utilize the correlations between labels in a sequence neighborhood to predict the best label sequence for a given input. As mentioned earlier, the task of the CRF layer is to predict one of three tags for each word of the input sequence: Action, Object, or None. The input to the CRF layer is the sequence from the attention layer. represents a certain output label sequence for , and represents the possible set of label sequences. The conditional probability function for the CRF, , over all possible label sequences given input sequence is given by:
where are potential functions to be learned. and are weight and bias matrices corresponding to the label pair . In this work, we use linear chain CRFs trained via maximum conditional log-likelihood estimation.
5.3.1. Enhancing CRFs via Additional Constraints
The Viterbi algorithm (Forney, 1973) used for decoding the CRF layer only considers interactions between sequentially adjacent tag labels. However, we encounter additional restrictions or constraints in our problem. First, we want to ensure that the CRF never predicts only an Action tag or an Object tag, since our definition mandates the occurrence of both an action and the object it acts upon to constitute a valid intent. Next, it is often useful to identify intent indicator phrases that suggest the presence of an intent in the corresponding text, or are characteristic of an action following them. Since it is challenging to construct a comprehensive list of all such intent indicators, we pick a small number of highly indicative phrases (Gupta et al., 2014; Wang et al., 2015) (see Section 6.2.1 for examples). For each such phrase, we selectively choose candidates having labelled intent tags in a small contextual neighbourhood following the intent indicator. In this work, we use up to five words as the neighborhood length.
These constraints operate at the level of the fully inferred sequence, and cannot be easily integrated into the Viterbi decoding algorithm by straightforward techniques like modifying its transition matrix (Kristjansson et al., 2004). We circumvent this in two ways during the tag inference phrase of the CRF. The first is to use a beam search based approach, which penalizes the sequences in the beam not satisfying the aforementioned constraints, and falls back to using the next most probable tag predictions. In the second approach, we adopt the idea of reducing the Viterbi decoding algorithm to a graph shortest path problem, solvable by Integer Linear Programming (Roth and Yih, 2005). The authors map a sequence of length with possible tag labels to a graph with nodes and edges. We extend this formulation by expressing our constraints as linear inequalities, and use this as the decoding algorithm for our CRF layer. The performance of these enhancements is shown in Table 3.
5.4. Generating Intents from Tag Sequences
Once the CRF predicts Action, Object and None tags for each word in the input text, our final step is to pair up and match the corresponding Action and Object tag phrases to generate coherent and meaningful user intents. As specified earlier, we define an intent as a combination of Action tagged phrases followed by Object tagged phrases. We develop two techniques for this purpose. First, we employ the simple but highly effective technique of linking Action and Object tagged phrases with respect to their word-based proximity in the input text. We assume that related action-object phrases are likely to occur spatially close to each other in this distance-based heuristic. For instance, in our earlier example, the action ‘reserve’ is more likely to match with the object seat’, than with the object ‘special meal’. However, this assumption may not always hold true, depending on the way the text has been worded.
This brings us to our second technique of matching appropriate Action-Object tagged phrases, by learning a multi-layer perceptron (MLP) classifier. The input features for the MLP consist of the sum of the pre-trained GloVe embeddings (alluded to in Section 4) of the words in the potential Action-Object intent phrase, concatenated with the normalized value of the spatial word distance between the Action and Object phrases in the original input text. These features account for both the word proximity of the constituent intent terms, and their semantic likelihood of co-occurring in a single phrase. The input to the MLP during both training and testing is thus the feature representation of all possible paired up combinations of the predicted Action and Object tagged phrases. On top of the input layer, the MLP contains two fully connected hidden layers of rectified linear units, followed by a fully connected layer of size one. This outputs a score for each potential Action-Object pair under consideration, indicating the likelihood of combining them to produce an intent. We train the MLP with a margin-based hinge loss function , that maximizes the separation margin between the true and the highest scoring incorrect Object phrase option for the current Action phrase.
where is the intended output of the MLP classifier, indicating whether the -th Action-Object pair is a correct match or not, and is the raw output value of its decision function. We present the performance of both the above techniques in Table 3.
Our self-attention based Bi-LSTM-CRF open intent extraction model thus makes use of semantic information from the previous and future time steps, and dependency constraints learned and enforced by the CRF; to predict intents for an input text utterance. Multi-head self-attention enables it to learn dependencies between distant words, possibly across sentences, effectively. Adversarial training acts as a powerful regularizer for our model, contributing to its robustness and resilience to user intents from diverse domains.
6.1. Data Collection
We collected about 75K questions with their top correct answer, on various topical categories and task domains, posed by users on the question-answer forum www.stackexchange.com. We then formulated an Amazon Mechanical Turk crowd sourcing experiment to annotate 25K of these questions with up to three intents (specifically, action phrases and their corresponding object phrases), that the crowd workers felt were most important or relevant.
|Name of Genre||No. of utterances||Avg length of utterance||Vocab size per genre|
|CRM software (name omitted for blind review)||11723||60||47219|
|Random forest classifier (features from Section 6.2.1)||0.66|
|MLP classifier (features from Section 6.2.1)||0.74|
|TOP-ID (stage I) classifier||0.91|
For the first stage of TOP-ID, i.e. recognizing the existence of an intent in an utterance, we generate two classes from our labeled intent data to provide as training data to the binary classifier. As the positive class (i.e. containing an intent), we use the questions from Stack Exchange that were labeled with user intents by the crowd workers. We assume for the purposes of this work that only the questions asked by individuals contain an actionable intent or task that a user desires or requests to be performed. The answers to these questions largely contain informative descriptions, reasons for specific events and occurrences, or methods to carry out activities. The authors of such answers often refer to task actions that need to be performed by others, while we seek to identify the personal intentions of users themselves. Hence, we use these answers as instances of the negative class, i.e. text inputs not containing any intent. This method of selecting the training and test data contributes to the robustness and efficacy of our binary classifier by minimizing the semantic gap in the topics constituting the two classes. It also ensures that our classifier correctly learns semantic, domain-independent indicators of the presence of an intent.
For the second stage of our TOP-ID framework, we use the remaining 50K unlabeled questions for the purpose of unsupervised pre-training. We first generate verb and object semantic parsing tags for these texts via the Stanford CoreNLP dependency parser (Manning et al., 2014). We employ the words tagged as verbs and objects as proxies for the Action and Object tagged phrases that compose an intent. We then fine-tune our model by further training it with the intent-labeled data, tagged with Action and Object phrases by our annotators.
In Table 1, we depict the different genres of our annotated Stack Exchange dataset. The last row shows a commercial Customer Relationship Management (CRM) software, whose name we omit for the purposes of the double-blind review. For each genre, we show the number of input question-answer texts collected, the average length of a question text, and the genre’s vocabulary size. Each question text serves as input to the second stage of our framework. There are hundreds of unique intent types per category. We choose Stack Exchange as our data source because of its long and verbose questions with background details and multiple intent actions scattered throughout the text. This is unlike the commonly used SNIPS (Coucke et al., 2018) and ATIS (Hemphill et al., 1990; Dahl et al., 1994) datasets for intent detection. These contain relatively shorter and crisper text on pre-defined intent categories, such as booking travels and weather inquiries. Having said that, we do analyze TOP-ID’s performance on the SNIPS dataset in Section 6.3.
6.2.1. TOP-ID Stage I: Open Intent Existence Prediction
Table 2 depicts the performance of the first stage of TOP-ID, i.e., discovering the existence of an intent. Two simple yet effective baselines we compared against include a random forest and an MLP classifier with two hidden layers of 100 ReLU units each. We framed their input features to capture various lexical and syntactic properties of natural language (Qadir and Riloff, 2011). These include: (i) the number of nouns and verbs in the text, (ii) does the utterance end in a noun or adjective, (iii) does the utterance begin with a verb or modal (e.g. will, would, could etc) word, (iv) the count of ‘Wh-’ markers (e.g. who, what, how etc) and question marks that signify a question, (v) does the text utterance contain personal pronouns, (vi) is there a first-person pronoun (e.g. i, we) within a three-word window of an infinitive verb phrase (‘to’ followed by a verb) in the utterance; and (vii) are there phrases indicating an ‘action plan’, or intent (e.g. plan to, want to, or would like to) – also used in Section 5.3.1.
Additional baselines include sigmoid classifiers deriving features from a CNN with 2D convolution and pooling operations (Kim, 2014), and RNNs (a GRU (Tang et al., 2015), LSTM and a Bi-LSTM) with a single hidden layer, max pooling and dropout. The first stage of TOP-ID achieves an F1-score above 90%, outperforming the baselines by at least 5%.
|Approach||ACTION P/R/F1||OBJECT P/R/F1||Intent P/R/F1||Semantic similarity|
|Phrases following intent indicators (from Section 5.3.1)||0.65/0.59/0.62||0.6/0.54/0.57||0.63/0.56/0.59||0.67|
|Stanford CoreNLP dependency parser (SC)||0.56/0.49/0.52||0.51/0.43/0.47||0.53/0.45/0.49||0.59|
|TOP-ID (att+adv) + train on SC + w-dist||0.60/0.53/0.56||0.57/0.47/0.52||0.58/0.49/0.53||0.70|
|TOP-ID (att+adv) + train on mturk + w-dist||0.75/0.59/0.66||0.74/0.52/0.61||0.74/0.55/0.63||0.74|
|TOP-ID (att) + pretrain on SC + fine tune on mturk + w-dist||0.78/0.62/0.69||0.79/0.56/0.66||0.78/0.58/0.67||0.80|
|TOP-ID (adv) + pretrain + fine tune + w-dist||0.81/0.60/0.68||0.76/0.54/0.63||0.78/0.56/0.65||0.77|
|TOP-ID (att+adv) + pretrain + fine tune + w-dist||0.84/0.66/0.73||0.81/0.63/0.71||0.82/0.64/0.72||0.83|
|TOP-ID (att+adv+beam-CRF) + pretrain + fine tune + w-dist||0.84/0.70/0.76||0.81/0.67/0.73||0.82/0.68/0.74||0.84|
|TOP-ID (att+adv+constr-CRF) + pretrain + fine tune + w-dist||0.84/0.72/0.77||0.81/0.67/0.73||0.82/0.69/0.75||0.85|
|TOP-ID (att+adv) + pretrain + fine tune + MLP||0.84/0.68/0.75||0.81/0.67/0.73||0.82/0.67/0.74||0.84|
|TOP-ID (att+adv+beam-CRF) + pretrain + fine tune + MLP||0.84/0.72/0.77||0.81/0.69/0.75||0.82/0.70/0.76||0.86|
|TOP-ID (att+adv+constr-CRF) + pretrain + fine tune + MLP||0.84/0.73/0.78||0.81/0.68/0.74||0.82/0.70/0.76||0.86|
6.2.2. TOP-ID Stage II: Open Intent Extraction
Baselines: Table 3 presents a comparison between various approaches for the task of open intent extraction, i.e stage II of our framework. In the first approach (first row), we simply return as intents the five-word phrases following the occurrence of any intent-indicator phrases (described in Sections 5.3.1 and 6.2.1). The second baseline is the verb-object tuples learned by the Stanford CoreNLP dependency parser (Manning et al., 2014), which we use as proxies for Action and Object tagged phrases respectively. The subsequent rows comprise different variants of TOP-ID. ‘att’ and ‘adv’ indicate the presence of attention and adversarial training respectively (from Section 5). ‘train on SC’ denotes model training only on verb-object tags obtained from the dependency parser. ‘train on mturk’ indicates model training only on the crowd worker annotated intent data. Except for the third and fourth rows of Table 3, all other variants of TOP-ID are first pre-trained on the verb-object tags learned by the dependency parser, followed by fine-tuning on the intent annotated utterances. ‘beam-CRF’ and ‘constr-CRF’ refer to the two CRF enhancing strategies from Section 5.3.1 of (i) considering a beam of probable tag sequences, and (ii) incorporating additional constraints into the CRF decoding algorithm. ‘w-dist’ and ‘MLP’ refer to the two techniques from Section 5.4 of matching appropriate Action-Object phrases to create a holistic intent based on (i) word proximity in the input text, and (ii) the score learned by the MLP classifier.
We reiterate that TOP-ID cannot be directly compared with existing intent detection techniques (e.g. (Hakkani-Tur et al., 2016; Liu and Lane, 2016; Zhang et al., 2018; Xia et al., 2018)) since these are formulated as classification problems, and classify intents into limited, pre-defined categories. They also require sufficient labeled training data for most (if not all) categories. Contrarily, TOP-ID handles thousands of distinct intent classes, and has no restrictions on the number of training examples needed per unique intent type.
Performance Metrics: We use the precision, recall, F1-score and semantic similarity metrics to evaluate the approaches in Table 3. The second column only considers the prediction performance of the Action tags for each word of the input utterance, whereas the third column only assesses the Object tags. The fourth column displays the results considering the combination of both tag types to create an intent. The last column of semantic similarity computes the average of the cosine similarities between the embeddings of the predicted and actual (ground truth) intents. The embedding for each predicted and true intent phrase is acquired by averaging the pre-trained GloVe embeddings (Pennington et al., 2014) of their constituent words. We ignore the words whose embeddings do not exist.
We observe a significant improvement of TOP-ID of over 15% in terms of F1-score and semantic similarity, compared to the simple intent-indicator based model and the Stanford parser (first two rows of Table 3). Utilizing the dependency parser data as a pre-training step for the weights of our model, followed by continuing the training on the actual intent-labeled data improves the F1-score by at least 6%. Enhancing the CRF decoding algorithm with added constraints (beam-CRF and constr-CRF) benefits the F1-score further by 2-5%. We find a performance difference of 3% between using the word proximity heuristic (w-dist), and the MLP classifier for matching Action and Object phrases. Overall, TOP-ID trained with attention, adversarial training and CRF enhancements outperforms all baselines in Table 3, with an intent F1 score of 76%, and a semantic similarity of 86% between the true and predicted intents.
6.2.3. Capacity of Domain Adaptation
|Test Domain Name||Intent F1||Intent F1+||Sim||Sim+|
|CRM software (name omitted)||0.79||0.83||0.88||0.91|
Encountering newly emergent niche domains or genres for which little to no labeled intent utterances are available is a fairly common real-world scenario. However, it is a time-consuming and labor-intensive process to obtain sufficient domain-specific annotated data for the purpose of model training and development. It is thus desirable to adapt and generalize an existing trained model with minimum re-training effort, each time a new domain with potentially new intents is added. We investigate the capability of our TOP-ID open intent discovery framework in adapting and transferring knowledge across distinct conversational domains or subjects.
In Table 4, we consider several different test domains. We train TOP-ID on utterances from the remaining domains other than the test domain. The second and fourth columns assess our method’s F1-score and semantic similarity, in predicting the correct intents per unique domain. The definitions of these metrics are the same as in Section 6.2.2. The third and fifth columns of Table 4 indicate the respective F1-score and semantic similarity achievable for the given test domain, when TOP-ID is trained using labeled data from the testing domain as well. The difference in both metrics with and without using training data from the test domain is , for most domain topics. Only the Life Hacks domain suffers a loss of 6.5% in terms of F1-score when we eliminate the data from this domain while training our model. Interestingly for the domain of DIY, its training data is dominated by other semantically distinct domains, as can be seen in Table 4. However, TOP-ID still attains a good F1-score of 72%, only 4% lesser than what was possible if DIY domain data was included in the training set. These results show that TOP-ID can easily and effectively be generalized to low-resource domains with minimal manual effort, to detect actionable intents in newly emerging domains with potentially novel intents.
6.2.4. Effect of Human-Annotated Training Data Size
In the third and fourth rows of Table 3, we showed that training our TOP-ID model with absolutely no human-labeled intent data is detrimental to its performance. We next examine in more detail, the effect of using varying amounts of human annotated intent data while training our model. Note that we also use the tags from the dependency parser to pre-train our model. Figure 4 shows the F1-score and semantic similarity values for the predicted intents achieved by TOP-ID, as the number of human annotated training instances varies. Both metrics are represented on the dual y-axes, by the blue and red plots respectively. We find that both plots are monotonically increasing. When the number of human labeled training instances across various domains is less than , both metrics are below 50%. The F1-score and semantic similarity rise to about 70% and 75% respectively at annotated training examples. Since we test TOP-ID on our annotated dataset that has distinct domain topics, this translates to slightly more than labeled examples per unique domain. Beyond this point, there is a steady performance improvement, with a less sharper gain than earlier. These observations reinforce TOP-ID’s domain adaptation capability and show that it does not require a large number of human labeled examples to attain a high performance.
6.2.5. Role of Attention
|Input Text Utterance||Intents|
|Is it possible to navigate back in <XXX> to previous page after save processing? … I have a page where I click on a link and use navigateURL … want to be able to go back to the previous calling page and complete the processing of the save…||navigate previous page, complete processing save|
|The "Your tweets retweeted" page on Twitter… find out all the users who retweeted a tweet of mine? … how many people have retweeted a tweet and what their Twitter IDs are?||find retweeted Twitter IDs|
|Is there a WordPress plugin that will tweet when a scheduled post is posted? I know there are tons … that will tweet when you publish a post, but none I have tried will do it on a scheduled post.||tweet when publish scheduled post|
|How can I keep my phone from just falling overwhen watching videos? … also want to have my hands free to doother things … but I find due to most phones not being particularly ‘grippy’ it is hard to lean them up … any nifty life hacks for this?||keep phone from falling, have hands free|
|I’m starting a micro-school… I want to manage sick notes and absences … How can I synchronize one central Google Calendar that only administrators have access… Parents should be able to schedule future absences and excuse past absences…||manage sick notes, manage absences, synchronize central calendar|
Table 3 indicates that the presence of attention lends TOP-ID an F1 score gain of at least 4%. We further explore TOP-ID’s capability of identifying relevant and meaningful semantic features from its input utterances, which contribute in discovering open intents. We examine and visualize in Table 5 the self-attention values for specific utterances from our Stack Exchange dataset. For the sake of brevity we display truncated versions of the text inputs in the first column, and the second column shows their associated user intents. A darker colored highlight on a specific utterance word indicates that it receives higher attention, and consequently plays a greater role in TOP-ID’s decision of intent discovery. Input utterance words that constitute intents are marked in boldface. In all cases, we observe that words semantically related to and contributing to at least one user intent are successfully identified by an attention head. For instance, the second row of Table 5 demonstrates the significance of ‘find out’, ‘retweeted’, ‘tweet’ and ‘what their Twitter IDs are’ in deciding the user intent of “find retweeted Twitter IDs". The attention heads are attentive to intent indicator phrases that are likely to precede an actionable intention, such as ‘possible to’, ‘want to be able to’, ‘how can I’ and ‘I want to’. Words that may represent an action or an object but are irrelevant to the user’s intent (e.g. ‘click on a link’ and ‘use navigateURL’ in the first row, ‘I find’ and ‘lean them up’ in the last row) also do not receive a high attention score. Further, our attention mechanism can capture the dependency between distant intent words, such as ‘find’ and ‘retweeted’ in the second row and ‘publish’ and ‘scheduled’ in the fourth row. It also associates the action ‘manage’ with two objects, ‘sick notes’ and ‘absences’, generating the intents “manage sick notes" and “manage absences".
6.3. Case Studies
We now test the efficacy of TOP-ID on two additional real-world datasets. Note that these serve as test instances for our models that have already been trained on our curated Stack Exchange dataset.
6.3.1. SNIPS NLU Benchmark Dataset
|User: Can someone please help? I’m trying to fix a broken ubuntu.|
|Agent: … how did you break it?|
|User: i’m on the cd and i’m trying to mount and then chroot my hd, which worked fine. I installed some new libs and now it no longer reboots.|
|User: what’s the easiest way to get a working boot on my drive again?|
|Agent: … sounds like something might be screwed up in your /etc/apt/sources.list file, if it’s failing on apt-get update|
|User: how can i fix my sources.list file?|
|Agent: open /etc/apt/sources.list. see if you notice any obvious errors|
|User: a question on the mounting issue - when i loaded the cd, my local hard drive was mounted in media, can’t i just use that as the chroot?|
|Agent: … assuming your flgrx is hosed, move the x conf file out of the way so that the radeon driver will be used instead …|
|User: … what do you suggest for a good backup program for ubuntu?|
|User: … i installed the latest radeon drivers manually. how do i upgrade to the newest kernel and default radeon drivers?|
|Agent: first you’d uninstall 10.6 fglrx driver. then you’d grab the three 2.6.34 deb packages and then install xorg-edgers repo. run grub-update so it finds the new kernel and done.|
|User: where do i get the debs? and i know how to uninstall the fglrx drivers …, and then do i copy back the xorg.conf.original to xorg.conf?|
|User: … do i need to add a source to my source list?|
|Agent: yes you need xorg-edgers (google it)|
|User: ok cool. how do i get rid of xorg, or is that already done?|
|Agent: … if you used jockey-gtk to install fglrx and no other method, then you should be able to use the same method to remove them|
This is a collection of over 16K crowdsourced queries on seven different topics provided by the commercial company SNIPS, and is widely used to benchmark the performance of automated dialog response agents (Coucke et al., 2018). This dataset has a vocabulary size nearly times smaller than our curated Stack Exchange dataset. The length of its average input is at least times shorter. Further, the SNIPS data is both linguistically and semantically less diverse as well as less complex due to the specificity of its constituent topics. We test TOP-ID trained on completely unrelated domains from Table 1 on the SNIPS dataset. We show in Figure 5 sample intents it discovers for the four distinct intent categories of PlayMusic, SearchCreativeWork, SearchScreeningEvent and BookRestaurant. The length of the bars represents the relative frequency of that particular intent in the input data. We observe that our method can be highly beneficial in drilling down further into the high-level intent categories, and understanding and summarizing the exact and specific fine-grained user actionable intents that they comprise of. For instance, in Figure 5(a), TOP-ID not only identifies the basic intents of ‘hear song’ or ‘play album’ in the PlayMusic category; but also tells us that users are interested in singers such as Leroi Moore, Eddie Vinson and James Iha, song albums like Curtain Call or concerto, and music platforms like Youtube and Zvooq. Further, though the focus of TOP-ID is open intent discovery and not slot filling, in most cases TOP-ID can automatically identify important and meaningful accompanying information apart from the user’s principal intent. For example, in Figure 5(c), a user’s overall desire is to search for screenings of a particular event. Our method accurately predicts this via keywords like search, locate, find and look. Moreover, TOP-ID also provides added information on the specific events that need to be searched, such as the Chump Change saga and the movie Heart Beat.
6.3.2. Ubuntu Dialog Corpus Chat Logs
Table 6 shows a real-world, multi-turn conversation between a user with technical issues (called User), and another who helps resolve them (called Agent). It belongs to the Ubuntu Dialog Corpus (Lowe et al., 2015). This dataset contains about one million technical support conversations related to the Ubuntu Linux operating system, and highly resembles real-world dialog exchanges with commercial customer care agents. The original dialog from which this snippet has been truncated contains more than turns. In general, such data is asynchronous with several dialog turns. It has diverse and informal user intents, dialog domains and semantic slots; which increases the difficulty of the open intent discovery task.
Note that TOP-ID has been trained on primarily unrelated genres from our Stack Exchange dataset, before testing on this dialog. Our goal here is to understand the intents of the user requesting support (User), and not the one providing it (Agent). The words constituting intents inferred by TOP-ID have been highlighted in boldface. We observe that though our framework was trained on labeled data which had up to three annotated intents, it is capable of recognizing more than three intents where applicable. TOP-ID recognizes the following user intents in the whole conversation: fix broken ubuntu, mount hd, chroot hd, get working boot, fix sources.list file, upgrade newest kernel, upgrade radeon drivers, get debs, uninstall fglrx drivers, copy xorg.conf original and get rid xorg. Using a classification-based intent detection approach, it would be quite difficult to meaningfully categorize such a conversation into a single intent type or category. Our TOP-ID framework on the other hand provides a realistic, fine-grained summary of the action items that a user intends to perform throughout the conversation.
In this work, we introduced and tackled the problem of open intent discovery. We developed a two-stage novel sequence tagging approach, TOP-ID, in contrast to the common method of modeling intent detection as a multi-class classification task. Our proposed framework harnesses a Bi-LSTM and a CRF coupled with self-attention and adversarial training. It can extract from user utterances multiple actionable intent types in a consistent format, many of which may be unseen during training. We additionally curated a large collection of 25K instances from diverse domains on Stack Exchange, and annotated them for general-purpose intents via crowd sourcing. Experiments and case studies on real-world datasets showed substantial improvements of our approach over competitive baselines. We also demonstrated TOP-ID’s ability to generalize and adapt across multiple domains, thereby minimizing the amount of labeled training data for a new task domain. TOP-ID provides an in-depth, fine-grained understanding of users’ prospective actions and intentions from their text utterances, which can greatly benefit downstream end-to-end conversational applications.
Our current work extracts explicitly mentioned user intents from their utterances. Promising future directions could include learning generative models for intents, and inferring implicitly present open intents from users’ text or speech. Another interesting direction could be applying TOP-ID to informal social media conversations.
- Agarwal and Sureka (2017) Swati Agarwal and Ashish Sureka. 2017. Characterizing Linguistic Attributes for Automatic Classification of Intent Based Racist/Radicalized Posts on Tumblr Micro-Blogging Website. arXiv preprint arXiv:1701.04931 (2017).
- Bhargava et al. (2013) Aditya Bhargava, Asli Celikyilmaz, Dilek Hakkani-Tür, and Ruhi Sarikaya. 2013. Easy contextual intent prediction and slot detection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
- Cai et al. (2017) Ruichu Cai, Binjun Zhu, Lei Ji, Tianyong Hao, Jun Yan, and Wenyin Liu. 2017. A CNN-LSTM Attention Approach to Understanding User Query Intent from Online Health Communities. In IEEE International Conference on Data Mining Workshops (ICDMW).
- Chen et al. (2017) Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. arXiv preprint arXiv:1707.05436 (2017).
- Chen et al. (2013) Zhiyuan Chen, Bing Liu, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh. 2013. Identifying intention posts in discussion forums. In North American chapter of the Association for Computational Linguistics: Human Language Technologies.
- Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190 (2018).
- Dahl et al. (1994) Deborah A Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Proceedings of the workshop on Human Language Technology.
- Forney (1973) G David Forney. 1973. The viterbi algorithm. Proc. IEEE 61, 3 (1973).
- Ganin and Lempitsky (2014) Yaroslav Ganin and Victor Lempitsky. 2014. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014).
- Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics.
- Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. Slot-Gated Modeling for Joint Slot Filling and Intent Prediction. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
- Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian E Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR).
- Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
- Gupta et al. (2014) Vineet Gupta, Devesh Varshney, Harsh Jhamtani, Deepam Kedia, and Shweta Karwa. 2014. Identifying Purchase Intent from Social Posts.. In ICWSM.
- Hakkani-Tur et al. (2016) Dilek Hakkani-Tur, Gokhan Tur, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM. In Proceedings of Interspeech.
- He et al. (2017) Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the Association for Computational Linguistics.
- Hemphill et al. (1990) Charles Hemphill, John Godfrey, and George Doddington. 1990. The ATIS spoken language systems pilot corpus. In a Workshop on Speech and Natural Language.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation (1997).
- Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015).
- Im and Cho (2017) Jinbae Im and Sungzoon Cho. 2017. Distance-based Self-Attention Network for Natural Language Inference. arXiv preprint arXiv:1712.02047 (2017).
- Jochim and Deleris (2017) Charles Jochim and Lea Deleris. 2017. Named Entity Recognition in the Medical Domain with Constrained CRF Models. In Proceedings of the European Chapter of the Association for Computational Linguistics.
- Kim et al. (2016) Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In IEEE Spoken Language Technology Workshop (SLT).
- Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014).
- Kim et al. (2017a) Young-Bum Kim, Sungjin Lee, and Karl Stratos. 2017a. Onenet: Joint domain, intent, slot prediction for spoken language understanding. In Automatic Speech Recognition and Understanding Workshop (ASRU).
- Kim et al. (2017b) Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017b. Adversarial adaptation of synthetic or stale data. In Proceedings of the Association for Computational Linguistics.
- Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
- Kristjansson et al. (2004) Trausti Kristjansson, Aron Culotta, Paul Viola, and Andrew McCallum. 2004. Interactive information extraction with constrained conditional random fields. In AAAI.
- Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001).
- Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 (2016).
- Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 (2017).
- Liu and Lane (2016) Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. arXiv preprint arXiv:1609.01454 (2016).
- Liu and Lane (2017) Bing Liu and Ian Lane. 2017. Multi-domain adversarial learning for slot filling in spoken language understanding. arXiv preprint arXiv:1711.11310 (2017).
- Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909 (2015).
- Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354 (2016).
- Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of the Association for Computational Linguistics.
- Mesnil et al. (2015) Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, et al. 2015. Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2015).
- Miyato et al. (2016) Takeru Miyato, Andrew Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725 (2016).
- Mrkšić et al. (2016) Nikola Mrkšić, Diarmuid O Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. arXiv preprint arXiv:1603.00892 (2016).
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of Empirical Methods in Natural Language Processing.
- Qadir and Riloff (2011) Ashequl Qadir and Ellen Riloff. 2011. Classifying sentences as speech acts in message board posts. In Proceedings of Empirical Methods in Natural Language Processing.
- Roth and Yih (2005) Dan Roth and Wen-tau Yih. 2005. Integer linear programming inference for conditional random fields. In Proceedings of the International Conference on Machine Learning.
- Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research (2014).
- Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387 (2015).
- Tan et al. (2017) Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2017. Deep semantic role labeling with self-attention. arXiv preprint arXiv:1712.01586 (2017).
- Tang et al. (2015) Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of Empirical Methods in Natural Language Processing.
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
- Wang et al. (2015) Jinpeng Wang, Gao Cong, Wayne Xin Zhao, and Xiaoming Li. 2015. Mining User Intents in Twitter: A Semi-Supervised Approach to Inferring Intent Categories for Tweets.. In AAAI.
- Xia et al. (2018) Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2018. Zero-shot User Intent Detection via Capsule Neural Networks. arXiv preprint arXiv:1809.00385 (2018).
- Xu and Sarikaya (2013) Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In Automatic Speech Recognition and Understanding (ASRU).
- Yu and Lam (2018) Qian Yu and Wai Lam. 2018. Product Question Intent Detection using Indicative Clause Attention and Adversarial Learning. In Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval.
- Zhang et al. (2018) Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S Yu. 2018. Joint Slot Filling and Intent Detection via Capsule Neural Networks. arXiv preprint arXiv:1812.09471 (2018).
- Zhang and Wang (2016) Xiaodong Zhang and Houfeng Wang. 2016. A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding.. In IJCAI.
- Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems.