Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
Abstract
The driving force behind the recent success of LSTMs has been their ability to learn complex and nonlinear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM’s final prediction. Using the phraselevel labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.
1 Introduction
In comparison with simpler linear models, techniques from deep learning have achieved impressive accuracy by effectively learning nonlinear interactions between features. However, due to our inability to describe the learned interactions, this improvement in accuracy has come at the cost of state of the art predictive algorithms being commonly regarded as blackboxes. In the domain of natural language processing (NLP), Long Short Term Memory networks (LSTMs) \citeplstm have become a basic building block, yielding excellent performance across a wide variety of tasks \citepseq2seq \citepsquad \citeplanguagemodels, while remaining largely inscrutable.
In this work, we introduce contextual decomposition (CD), a novel interpretation method for explaining individual predictions made by an LSTM without any modifications to the underlying model. CD extracts information about not only which words contributed to a LSTM’s prediction, but also how they were combined in order to yield the final prediction. By mathematically decomposing the LSTM’s output, we are able to disambiguate the contributions made at each step by different parts of the sentence.
To validate the CD interpretations extracted from an LSTM, we evaluate on the problem of sentiment analysis. In particular, we demonstrate that CD is capable of identifying words and phrases of differing sentiment within a given review. CD is also used to successfully extract positive and negative negations from an LSTM, something that has not previously been done. As a consequence of this analysis, we also show that prior interpretation methods produce scores which have documentlevel information built into them in complex, unspecified ways. For instance, prior work often identifies strongly negative phrases contained within positive reviews as neutral, or even positive.
2 Related Work
The most relevant prior work on interpreting LSTMs has focused on approaches for computing wordlevel importance scores, with evaluation protocols varying greatly. \citetmurdoch17 introduced a decomposition of the LSTM’s output embedding into a sum over word coefficients, and demonstrated that those coefficients are meaningful by using them to distill LSTMs into rulesbased classifiers. \citetloo took a more black box approach, called Leave One Out, by observing the change in log probability resulting from replacing a given word vector with a zero vector, and relied solely on anecdotal evaluation. Finally, \citetsundararajan presents a general gradientbased technique, called Integrated Gradients, which was validated both theoretically and with empirical anecdotes. In contrast to our proposed method, this line of work has been limited to wordbased importance scores, ignoring the interactions between variables which make LSTMs so accurate.
Another line of work \citepkarpathy \citephendrik has focused on analysing the movement of raw gate activations over a sequence. \citetkarpathy was able to identify some coordinates of the cell state that correspond to semantically meaningful attributes, such as whether the text is in quotes. However, most of the cell coordinates were uninterpretable, and it is not clear how these coordinates combine to contribute to the actual prediction.
Decompositionbased approaches to interpretation have also been applied to convolutional neural networks (CNNs) \citeplrp \citepdeeplift. However, they have been limited to producing pixellevel importance scores, ignoring interactions between pixels, which are clearly quite important. Our approach is similar to these in that it computes an exact decomposition, but we leverage the unique gating structure of LSTMs in order to extract interactions.
Attention based models \citepbahdanau offer another means of providing some interpretability. Such models have been successfully applied to many problems, yielding improved performance \citeprush2015summary \citepxu2015show. In contrast to other word importance scores, attention is limited in that it only provides an indirect indicator of importance, with no directionality, i.e. what class the word is important for. Although attention weights are often cited anecdotally, they have not been evaluated, empirically or otherwise, as an interpretation technique. As with other prior work, attention is also incapable of describing interactions between words.
3 Contextual Decomposition of LSTMs
Given an arbitrary phrase contained within an input, we present a novel decomposition of the output of an LSTM into a sum of two contributions: those resulting solely from the given phrase, and those involving other factors. The key insight behind this decomposition is that the gating dynamics unique to LSTMs are a vehicle for modeling interactions between variables.
3.1 Long Short Term Memory Networks
Over the past few years, LSTMs have become a core component of neural NLP systems. Given a sequence of word embeddings , a cell and state vector are computed for each element by iteratively applying the below equations, with initialization .
(1)  
(2)  
(3)  
(4)  
(5)  
(6) 
Where , , and denotes elementwise multiplication. and are often referred to as output, forget and input gates, respectively, due to the fact that their values are bounded between and , and that they are used in elementwise multiplication.
After processing the full sequence, the final state is treated as a vector of learned features, and used as input to a multinomial logistic regression, often called SoftMax, to return a probability distribution over classes, with
(7) 
3.2 Contextual Decomposition of LSTM
We now introduce contextual decomposition, our proposed method for interpreting LSTMs. Given an arbitrary phrase , where , we now decompose each output and cell state in Equations 5 and 6 into a sum of two contributions.
(8)  
(9) 
The decomposition is constructed so that corresponds to contributions made solely by the given phrase to , and that corresponds to contributions involving, at least in part, elements outside of the phrase. and represent analogous contributions to .
Using this decomposition for the final output state in Equation 7 yields
(10) 
Here provides a quantitative score for the phrase’s contribution to the LSTM’s prediction. As this score corresponds to the input to a logistic regression, it may be interpreted in the same way as a standard logistic regression coefficient.
Disambiguating interactions between gates
In the cell update Equation 5, neuron values in each of and are independently determined by both the contribution at that step, , as well as prior context provided by . Thus, in computing the elementwise product , often referred to as gating, contributions made by to interact with contributions made by to , and vice versa.
We leverage this simple insight to construct our decomposition. First, assume that we have a way of linearizing the gates and updates in Equations 2, 3, 4 so that we can write each of them as a linear sum of contributions from each of their inputs.
(11)  
(12) 
When we use this linearization in the cell update Equation 5, the products between gates become products over linear sums of contributions from different factors. Upon expanding these products, the resulting crossterms yield a natural interpretation as being interactions between variables. In particular, crossterms can be assigned as to whether they resulted solely from the phrase, e.g. , from some interaction between the phrase and other factors, e.g. , or purely from other factors, e.g. .
Mirroring the recurrent nature of LSTMs, the above insights allow us to recursively compute our decomposition, with the initializations . We derive below the update equations for the case where , so that the current time step is contained within the phrase. The other case is similar, and the general recursion formula is provided in Appendix A.2.
For clarity, we decompose the two products in the cell update Equation 5 separately. As discussed above, we simply linearize the gates involved, expand the resulting product of sums, and group the crossterms according to whether or not their contributions derive solely from the specified phrase, or otherwise. Terms are determined to derive solely from the specified phrase if they involve products from some combination of and or (but not both). When is not within the phrase, products involving are treated as not deriving from the phrase.
(13)  
(14)  
(15) 
(16)  
(17)  
(18) 
Having decomposed the two components of the cell update equation, we can attain our decomposition of by summing the two contributions.
(19)  
(20) 
Once we have computed the decomposition of , it is relatively simple to compute the resulting transformation of by linearizing the function in 6. Note that we could similarly decompose the output gate as we treated the forget gate above, but we empirically found this to not produce improved results.
(21)  
(22)  
(23)  
(24) 
Linearizing Activation Functions
We now describe the linearizing functions used in the above decomposition. Formally, for arbitrary , where , the problem is how to write
(25) 
In the cases where there is a natural ordering to , prior work \citepmurdoch17 has used a telescoping sum consisting of differences of partial sums as a linearization technique, which we show below.
(26) 
However, in our setting contains terms such as , and , which have no clear ordering. Thus, there is no natural way to order the sum in Equation 26. Instead, we compute an average over all orderings. Letting denote the set of all permutations of , our score is given below. Note that when , the corresponding term is equal to equation 26.
(27) 
can be analogously derived. When one of the terms in the decomposition is a bias, we saw improvements when restricting to permutations where the bias is the first term.
As only ranges between and , this linearization generally takes very simple forms. For instance, when , the contribution assigned to is
(28) 
This linearization was presented in a scalar context where , but trivially generalizes to the vector setting . It can also be viewed as an approximation to Shapely values, as discussed in \citetlundbergLee and \citetdeeplift.
4 Experiments
We now describe our empirical validation of CD on the task of sentiment analysis. First, we verify that, on the standard problem of wordlevel importance scores, CD compares favorably to prior work. Then we examine the behavior of CD for word and phrase level importance in situations involving compositionality, showing that CD is able to capture the composition of phrases of differing sentiment. Finally, we show that CD is capable of extracting instances of positive and negative negation. Code for computing CD scores is available online
4.1 Training Details
We first describe the process for fitting models which are used to produce interpretations. As the primary intent of this paper is not predictive accuracy, we used standard best practices without much tuning. We implemented all models in Torch using default hyperparameters for weight initializations. All models were optimized using Adam \citepadam with the default learning rate of 0.001 using early stopping on the validation set. For the linear model, we used a bag of vectors model, where we sum pretrained Glove vectors \citepglove and add an additional linear layer from the word embedding dimension, 300, to the number of classes, 2. We fine tuned both the word vectors and linear parameters. We will use the two data sets described below to validate our new CD method.
Stanford Sentiment Treebank
We trained an LSTM model on the binary version of the Stanford Sentiment Treebank (SST) \citepsst, a standard NLP benchmark which consists of movie reviews ranging from 2 to 52 words long. In addition to reviewlevel labels, it also provides labels for each phrase in the binarized constituency parse tree. Following the hyperparameter choices in \citettreelstm, the word and hidden representations of our LSTM were set to 300 and 168, and word vectors were initialized to pretrained Glove vectors \citepglove. Our LSTM attains 87.2% accuracy, and we also train a logistic regression model with bag of words features, which attains 83.2% accuracy.
Yelp Polarity
Originally introduced in \citetcharcnn, the Yelp review polarity dataset was obtained from the Yelp Dataset Challenge and has train and test sets of sizes 560,000 and 38,000. The task is binary prediction for whether the review is positive (four or five stars) or negative (one or two stars). The reviews are relatively long, with an average length of 160.1 words. Following the guidelines from \citetcharcnn, we implement an LSTM model which attains 4.6% error, and an ngram logistic regression model, which attains 5.7% error. For computational reasons, we report interpretation results on a random subset of sentences of length at most 40 words. When computing integrated gradient scores, we found that numerical issues produced unusable outputs for roughly 6% of the samples. These reviews are excluded.
Interpretation Baselines
We compare the interpretations produced by CD against four state of the art baselines: cell decomposition \citepmurdoch17, integrated gradients \citepsundararajan, leave one out \citeploo, and gradient times input. We refer the reader to Section 2 for descriptions of these algorithms. For our gradient baseline, we compute the gradient of the output probability with respect to the word embeddings, and report the dot product between the word vector and its gradient. For integrated gradients, producing reasonable values required extended experimentation and communication with the creators regarding the choice of baselines and scaling issues. We ultimately used sequences of periods for our baselines, and rescaled the scores for each review by the standard deviation of the scores for that review, a trick not previously mentioned in the literature. To obtain phrase scores for wordbased baselines integrated gradients, cell decomposition, and gradients, we sum the scores of the words contained within the phrase. For the purposes of visualization, we convert the two dimensional vector of CD outputs into one dimension by taking the difference between the two scores. Empirically, we observed that this difference in scores is, up to a factor of two, essentially the same as either of the scores individually.
4.2 Unigram (word) scores
Before examining the novel, phraselevel dynamics of CD, we first verify that it compares favorably to prior work for the standard use case of producing unigram coefficients. When sufficiently accurate in terms of prediction, logistic regression coefficients are generally treated as a gold standard for interpretability. In particular, when applied to sentiment analysis the ordering of words given by their coefficient value provides a qualitatively sensible measure of importance. Thus, when determining the validity of coefficients extracted from an LSTM, we should expect there to be a meaningful relationship between the CD scores and logistic regression coefficients.
In order to evaluate the wordlevel coefficients extracted by the CD method, we construct scatter plots with each point consisting of a single word in the validation set. The two values plotted correspond to the coefficient from logistic regression and importance score extracted from the LSTM. For a quantitative measure of accuracy, we use pearson correlation coefficient.
We report quantitative and qualitative results in Appendix A.1.3. For SST, CD and integrated gradients, with correlations of 0.76 and 0.72, respectively, are substantially better than other methods, with correlations of at most 0.51. On Yelp, the gap is not as big, but CD is still very competitive, having correlation 0.52 with other methods ranging from 0.34 to 0.56. Having verified reasonably strong results in this base case, we now proceed to show the benefits of CD.
4.3 Identifying dissenting subphrases
We now show that, for phrases of at most five words, existing methods are unable to recognize subphrases with differing sentiments. For example, consider the phrase “used to be my favorite”, which is of negative sentiment. The word “favorite”, however, is strongly positive, having a logistic regression coefficient in the 93rd percentile. Nonetheless, existing methods consistently rank “favorite” as being highly negative or neutral. In contrast, as shown in Table 1, CD is able to identify “my favorite” as being strongly positive, and ”used to be” as strongly negative. A similar dynamic also occurs with the phrase “not worth the time”. The main justification for using LSTMs over simpler models is precisely that they are able to capture these kinds of interactions. Thus, it is important that an interpretation algorithm is able to properly uncover how the interactions are being handled.
Using the above as a motivating example, we now show that a similar trend holds throughout the Yelp polarity dataset. In particular, we conduct a search for situations similar to the above, where a strongly positive/negative phrase contains a strongly dissenting subphrase. Phrases are scored using the logistic regression with ngram features described in Section 4.1, and included if their absolute score is over 1.5. We then examine the distribution of scores for the dissenting subphrases, which are analogous to “favorite”.
For an effective interpretation algorithm, the distribution of scores for positive and negative dissenting subphrases should be significantly separate, with positive subphrases having positive scores, and vice versa. However, as can be seen in Appendix A.1.1, for prior methods these two distributions are nearly identical. The CD distributions, on the other hand, are significantly separate, indicating that what we observed anecdotally above holds in a more general setting.
\RaggedRight\arraybackslashAttribution Method  Heat Map  

\RaggedRight\arraybackslashGradient  \RaggedRight\arraybackslashused to be my favorite  \RaggedRight\arraybackslashnot worth the time 
\RaggedRight\arraybackslashLeave One Out \citeploo  \RaggedRight\arraybackslashused to be my favorite  \RaggedRight\arraybackslashnot worth the time 
\RaggedRight\arraybackslashCell decomposition \citepmurdoch17  \RaggedRight\arraybackslashused to be my favorite  \RaggedRight\arraybackslashnot worth the time 
\RaggedRight\arraybackslashIntegrated gradients \citepsundararajan  \RaggedRight\arraybackslashused to be my favorite  \RaggedRight\arraybackslashnot worth the time 
\RaggedRight\arraybackslashContextual decomposition  \RaggedRight\arraybackslashused to be my favorite  \RaggedRight\arraybackslashnot worth the time 

Legend Very Negative Negative Neutral Positive Very Positive
4.4 Examining highlevel compositionality
We now show that prior methods struggle to identify cases where a sizable portion of a review (between one and two thirds) has polarity different from the LSTM’s prediction. For instance, consider the review in Table 2, where the first phrase is clearly positive, but the second phrase causes the review to ultimately be negative. CD is the only method able to accurately capture this dynamic.
By leveraging the phraselevel labels provided in SST, we can show that this pattern holds in the general case. In particular, we conduct a search for reviews similar to the above example. The search criteria are whether a review contains a phrase labeled by SST to be of opposing sentiment to the reviewlevel SST label, and is between one and two thirds the length of the review.
In Appendix A.1.2, we show the distribution of the resulting positive and negative phrases for different attribution methods. A successful interpretation method would have a sizable gap between these two distributions, with positive phrases having mostly positive scores, and negative phrases mostly negative. However, prior methods struggle to satisfy these criteria. 87% of all positive phrases are labelled as negative by integrated gradients, and cell decompositions \citepmurdoch17 even have the distributions flipped, with negative phrases yielding more positive scores than the positive phrases. CD, on the other hand, provides a very clear difference in distributions. To quantify this separation between positive and negative distributions, we examine a twosample KolmogorovSmirnov onesided test statistic, a common test for the difference of distributions with values ranging from to . CD produces a score of 0.74, indicating a strong difference between positive and negative distributions, with other methods achieving scores of 0 (cell decomposition), 0.33 (integrated gradients), 0.58 (leave one out) and 0.61 (gradient), indicating weaker distributional differences. Given that gradient and leave one out were the weakest performers in unigram scores, this provides strong evidence for the superiority of CD.
Attribution Method  Heat Map 

Gradient  It’s easy to love Robin Tunney – she’s pretty and she can act – but it gets harder and harder to understand her choices. 
Leave one out \citeploo  It’s easy to love Robin Tunney – she’s pretty and she can act – but it gets harder and harder to understand her choices. 
Cell decomposition \citepmurdoch17  It’s easy to love Robin Tunney – she’s pretty and she can act – but it gets harder and harder to understand her choices. 
Integrated gradients \citepsundararajan  It’s easy to love Robin Tunney – she’s pretty and she can act – but it gets harder and harder to understand her choices. 
Contextual decomposition  It’s easy to love Robin Tunney – she’s pretty and she can act – but it gets harder and harder to understand her choices. 

Legend Very Negative Negative Neutral Positive Very Positive
4.5 Contextual decomposition (CD) captures negation
In order to understand an LSTM’s prediction mechanism, it is important to understand not just the contribution of a phrase, but how that contribution is computed. For phrases involving negation, we now demonstrate that we can use CD to empirically show that our LSTM learns a negation mechanism.
Using the phrase labels in SST, we search over the training set for instances of negation. In particular, we search for phrases of length less than ten with the first child containing a negation phrase (such as “not” or “lacks”, full list provided in Appendix A.3) in the first two words, and the second child having positive or negative sentiment. Due to noise in the labels, we also included phrases where the entire phrase was nonneutral, and the second child contained a nonneutral phrase. We identify both positive negation, such as “isn’t a bad film”, and negative negation, such as “isn’t very interesting”, where the direction is given by the SSTprovided label of the phrase.
For a given negation phrase, we extract a negation interaction by computing the CD score of the entire phrase and subtracting the CD scores of the phrase being negated and the negation term itself. The resulting score can be interpreted as an ngram feature. Note that, of the methods we compare against, only leave one out is capable of producing such interaction scores. For reference, we also provide the distribution of all interactions for phrases of length less than 5.
We present the distribution of extracted scores in Figure 1. For CD, we can see that there is a clear distinction between positive and negative negations, and that the negation interactions are centered on the outer edges of the distribution of interactions. Leave one out is able to capture some of the interactions, but has a noticeable overlap between positive and negative negations around zero, indicating a high rate of false negatives.
4.6 Identifying similar phrases
Another benefit of using CDs for interpretation is that, in addition to providing importance scores, it also provides dense embeddings for arbitrary phrases and interactions, in the form of discussed in Section 3.2. We anecdotally show that similarity in this embedding space corresponds to semantic similarity in the context of sentiment analysis.
In particular, for all words and binary interactions, we compute the average embedding produced by CD across the training and validation sets. In Table 3, we show the nearest neighbours using a cosine similarity metric. The results are qualitatively sensible for three different kinds of interactions: positive negation, negative negation and modification, as well as positive and negative words. Note that we for positive and negative words, we chose the positive/negative parts of the negations, in order to emphasize that CD can disentangle this composition.
\RaggedRight\arraybackslashnot entertaining  \RaggedRight\arraybackslashnot bad  \RaggedRight\arraybackslashvery funny  \RaggedRight\arraybackslashentertaining  \RaggedRight\arraybackslashbad 

\RaggedRight\arraybackslashnot funny  \RaggedRight\arraybackslashnever dull  \RaggedRight\arraybackslashwellputtogether piece  \RaggedRight\arraybackslashintelligent  \RaggedRight\arraybackslashdull 
\RaggedRight\arraybackslashnot engaging  \RaggedRight\arraybackslashn’t drag  \RaggedRight\arraybackslashentertaining romp  \RaggedRight\arraybackslashengaging  \RaggedRight\arraybackslashdrag 
\RaggedRight\arraybackslashnever satisfactory  \RaggedRight\arraybackslashnever fails  \RaggedRight\arraybackslashvery good  \RaggedRight\arraybackslashsatisfying  \RaggedRight\arraybackslashawful 
\RaggedRight\arraybackslashnot well  \RaggedRight\arraybackslashwithout sham  \RaggedRight\arraybackslashsurprisingly sweet  \RaggedRight\arraybackslashadmirable  \RaggedRight\arraybackslashtired 
\RaggedRight\arraybackslashnot fit  \RaggedRight\arraybackslashwithout missing  \RaggedRight\arraybackslashvery wellwritten  \RaggedRight\arraybackslashfunny  \RaggedRight\arraybackslashdreary 
5 Conclusion
In this paper, we have proposed contextual decomposition (CD), an algorithm for interpreting individual predictions made by LSTMs without modifying the underlying model. In both NLP and general applications of LSTMs, CD produces importance scores for words (single variables in general), phrases (several variables together) and word interactions (variable interactions). Using two sentiment analysis datasets for empirical validation, we first show that for information also produced by prior methods, such as wordlevel scores, our method compares favorably. More importantly, we then show that CD is capable of identifying phrases of varying sentiment, and extracting meaningful word (or variable) interactions. This movement beyond wordlevel importance is critical for understanding a model as complex and highly nonlinear as LSTMs.
Acknowledgments
This research was started during a summer internship at Google Brain, and later supported by a postgraduate scholarshipdoctoral from NSERC and a data science research award from Adobe. This work is partially supported by Center for Science of Information (CSoI), an NSF Science and Technology Center, under grant agreement CCF0939370, ONR grant N000141612664 and ARO grant W911NF1710005.
Appendix A Appendix
a.1 Plots
Plots for dissenting subphrases
We provide here the plots described in Section 4.3.
Plots for highlevel compositionality
We provide here the plots referenced in Section 4.4.
Logistic regression versus extracted coefficients scatterplots
We provide here the scatterplots and correlations referenced in section 4.2.
\RaggedRight\arraybackslashAttribution Method  \RaggedRight\arraybackslashStanford Sentiment  \RaggedRight\arraybackslashYelp Polarity 

\RaggedRight\arraybackslashGradient  \RaggedRight\arraybackslash0.375  \RaggedRight\arraybackslash0.336 
\RaggedRight\arraybackslashLeave one out \citeploo  \RaggedRight\arraybackslash0.510  \RaggedRight\arraybackslash0.358 
\RaggedRight\arraybackslashCell decomposition \citepmurdoch17  \RaggedRight\arraybackslash0.490  \RaggedRight\arraybackslash0.560 
\RaggedRight\arraybackslashIntegrated gradients \citepsundararajan  \RaggedRight\arraybackslash0.724  \RaggedRight\arraybackslash0.471 
\RaggedRight\arraybackslashContextual decomposition  \RaggedRight\arraybackslash0.758  \RaggedRight\arraybackslash0.520 
a.2 General recursion formula
We provide here the general recursion formula referenced in Section 3.2.1. The two cases that are considered is whether the current time step is during the phrase () or outside of the phrase ( or ).
(29)  
(30)  
(31)  
(32)  
a.3 List of words used to identify negations
To search for negations, we used the following list of negation words: not, n’t, lacks, nobody, nor, nothing, neither, never, none, nowhere, remotely
Footnotes
 https://github.com/jamiemurdoch/ContextualDecomposition
References
 Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, KlausRobert Müller, and Wojciech Samek. On pixelwise explanations for nonlinear classifier decisions by layerwise relevance propagation. PloS one, 10(7):e0130140, 2015.
 Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
 Sepp Hochreiter and Jürgen Schmidhuber. Long shortterm memory. Neural computation, 9(8):1735–1780, 1997.
 Andrej Karpathy, Justin Johnson, and Li FeiFei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015.
 Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
 Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. CoRR, abs/1612.08220, 2016. URL http://arxiv.org/abs/1612.08220.
 Scott Lundberg and SuIn Lee. An unexpected unity among methods for interpreting model predictions. arXiv preprint arXiv:1611.07478, 2016.
 Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. CoRR, abs/1707.05589, 2017. URL http://arxiv.org/abs/1707.05589.
 W James Murdoch and Arthur Szlam. Automatic rule extraction from long short term memory networks. ICLR, 2017.
 Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014.
 Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
 Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015.
 Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685, 2017.
 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.
 Hendrik Strobelt, Sebastian Gehrmann, Bernd Huber, Hanspeter Pfister, and Alexander M Rush. Visual analysis of hidden state dynamics in recurrent neural networks. arXiv preprint arXiv:1606.07461, 2016.
 Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR, abs/1703.01365, 2017. URL http://arxiv.org/abs/1703.01365.
 Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014.
 Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from treestructured long shortterm memory networks. arXiv preprint arXiv:1503.00075, 2015.
 Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pp. 2048–2057, 2015.
 Xiang Zhang, Junbo Zhao, and Yann LeCun. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems, pp. 649–657, 2015.