Explaining and Interpreting LSTMs
Abstract
While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layerwise Relevance Propagation (LRP) technique used for explaining the predictions of feedforward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.
Keywords:
Explainable Artificial Intelligence Model Transparency Recurrent Neural Networks LSTM Interpretability1 Introduction
\@footnotetextThe final authenticated publication is available online at https://doi.org/10.1007/9783030289546_11. In: W. Samek et al. (Eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science, vol 11700, pp. 211238. Springer, Cham (2019)
In practical applications, building highperforming AI systems is not always the sole objective, and interpretability may also be an important issue [16].
Most of the recent research on interpretable AI has focused on feedforward neural networks, especially the deep rectifier networks and variants used for image recognition [68, 79]. Layerwise relevance propagation (LRP) [6, 51] was shown in this setting to provide for stateoftheart models such as VGG16, explanations that are both informative and fast to compute, and that could be embedded in the framework of deep Taylor decomposition [52].
However, in the presence of sequential data, one may need to incorporate temporal structure in the neural network model, e.g. to make forecasts about future time steps. In this setting it is key to be able to learn the underlying dynamical system, e.g. with a recurrent neural network, so that it can then be simulated forward. Learning dynamical systems with longterm dependencies using recurrent neural networks presents a number of challenges. The backpropagation through time learning signal tends to either blow up or vanish [30, 10]. To reduce this difficulty, special neural network architectures have been proposed, in particular, the Long ShortTerm Memory (LSTM) [30, 35, 37], which makes use of special accumulators and gating functions.
The multiple architectural changes and the unique nature of the sequential prediction task make a direct application of the LRPtype explanation technique nonstraightforward. To be able to deliver accurate explanations, one needs to carefully inspect the structure of the LSTM blocks forming the model and their interaction.
In this chapter, we explore multiple dimensions of the interface between the LRP technique and the LSTM. First, we analyze how the LRP propagation mechanism can be adapted to accumulators and gated interactions in the LSTM. Our new propagation scheme is embedded in the deep Taylor decomposition framework [52], and validated empirically on sentiment analysis and on a toy numeric task. Further, we investigate how modifications of the LSTM architecture, in particular, on the cell input activation, the forget and output gates and on the network connections, make explanations more straightforward, and we apply these changes in a reinforcement learning showcase.
2 Background
2.1 Long ShortTerm Memory (LSTM)
Recently, Long ShortTerm Memory (LSTM; [30, 35, 37]) networks have emerged as the bestperforming technique in speech and language processing. LSTM networks have been overwhelmingly successful in different speech and language applications, including handwriting recognition [24], generation of writings [23], language modeling and identification [22, 78], automatic language translation [73], speech recognition [63, 17], analysis of audio data [49], analysis, annotation, and description of video data [15, 76, 70]. LSTM has facilitated recent benchmark records in TIMIT phoneme recognition, optical character recognition, texttospeech synthesis, language identification, large vocabulary speech recognition, EnglishtoFrench translation, audio onset detection, social signal classification, image caption generation, videototext description, endtoend speech recognition, and semantic representations.
The key idea of LSTM is the use of memory cells that allow for constant error flow during training. Thereby, LSTM avoids the vanishing gradient problem, that is, the phenomenon that training errors are decaying when they are backpropagated through time [30, 33]. The vanishing gradient problem severely impedes credit assignment in recurrent neural networks, i.e. the correct identification of relevant events whose effects are not immediate, but observed with possibly long delays. LSTM, by its constant error flow, avoids vanishing gradients and, hence, allows for uniform credit assignment, i.e. all input signals obtain a similar error signal. Other recurrent neural networks are not able to assign the same credit to all input signals and therefore, are very limited concerning the solutions they will find. Uniform credit assignment enables LSTM networks to excel in speech and language tasks: if a sentence is analyzed, then the first word can be as important as the last word. Via uniform credit assignment, LSTM networks regard all words of a sentence equally. Uniform credit assignment enables to consider all input information at each phase of learning, no matter where it is located in the input sequence. Therefore, uniform credit assignment reveals many more solutions to the learning algorithm, which would otherwise remain hidden.
LSTM in a Nutshell.
The central processing and storage unit for LSTM recurrent networks is the memory cell. As already mentioned, it avoids vanishing gradients and allows for uniform credit assignment. The most commonly used LSTM memory cell architecture in the literature [25, 66] contains forget gates [19, 20] and peephole connections [18]. In our previous work [38, 34], we found that peephole connections are only useful for modeling time series, but not for language, metalearning, or biological sequences. That peephole connections can be removed without performance decrease, was recently confirmed in a large assessment, where different LSTM architectures have been tested [26]. While LSTM networks are highly successful in various applications, the central memory cell architecture was not modified since 2000 [66]. A memory cell architecture without peepholes is depicted in Figure 1.
In our definition of an LSTM network, all units of one kind are pooled to a vector: is the vector of cell input activations, is the vector of input gate activations, is the vector of forget gate activations, is the vector of memory cell states, is the vector of output gate activations, and is the vector of cell output activations.
We assume to have an input sequence, where the input vector at time is . The matrices , , , and correspond to the weights of the connections between inputs and cell input, input gate, forget gate, and output gate, respectively. The matrices , , , and correspond to the weights of the connections between the cell output activations with onestep delay and cell input, input gate, forget gate, and output gate, respectively. The vectors , , , and are the bias vectors of cell input, input gate, forget gate, and output gate, respectively. The activation functions are for the cell input, for the cell state, and for the gates, where these functions are evaluated in a componentwise manner if they are applied to vectors. Typically, either the sigmoid or are used as activation functions. denotes the pointwise multiplication of two vectors. Without peepholes, the LSTM memory cell forward pass rules are (see Figure 1):
cell input  (1)  
input gate  (2)  
forget gate  (3)  
cell state  (4)  
output gate  (5)  
cell output  (6) 
LongTerm Dependencies vs. Uniform Credit Assignment.
The LSTM network has been proposed with the aim to learn longterm dependencies in sequences which span over long intervals [37, 36, 31, 32]. However, besides extracting longterm dependencies, LSTM memory cells have another, even more important, advantage in sequence learning: as already described in the early 1990s, LSTM memory cells allow for uniform credit assignment, that is, the propagation of errors back to inputs without scaling them [30]. For uniform credit assignment of current LSTM architectures, the forget gate must be one or close to one. A memory cell without an input gate just sums up all the squashed inputs it receives during scanning the input sequence. Thus, such a memory cell is equivalent to a unit that sees all sequence elements at the same time, as has been shown via the “Ersatzschaltbild” (engl. equivalent circuit diagram) [30]. If an output error occurs only at the end of the sequence, such a memory cell, via backpropagation, supplies the same delta error at the cell input unit at every time step. Thus, all inputs obtain the same credit for producing the correct output and are treated on an equal level and, consequently, the incoming weights to a memory cell are adjusted by using the same delta error at the input unit .
In contrast to LSTM memory cells, standard recurrent networks scale the delta error and assign different credit to different inputs. The more recent the input, the more credit it obtains. The first inputs of the sequence are hidden from the final states of the recurrent network. In many learning tasks, however, important information is distributed over the entire length of the sequence and can even occur at the very beginning. For example, in language and textrelated tasks, the first words are often important for the meaning of a sentence. If the credit assignment is not uniform along the input sequence, then learning is very limited. Learning would start by trying to improve the prediction solely by using the most recent inputs. Therefore, the solutions that can be found are restricted to those that can be constructed if the last inputs are considered first. Thus, only those solutions are found that are accessible by gradient descent from regions in the parameter space that only use the most recent input information. In general, these limitations lead to suboptimal solutions, since learning gets trapped in local optima. Typically, these local optima correspond to solutions which efficiently exploit the most recent information in the input sequence, while information way back in the past is neglected.
2.2 LayerWise Relevance Propagation (LRP)
Layerwise relevance propagation (LRP) [6] (cf. [51] for an overview) is a technique to explain individual predictions of deep neural networks in terms of input variables. For a given input and the neural network’s prediction, it assigns a score to each of the input variables indicating to which extent they contributed to the prediction. LRP works by reversepropagating the prediction through the network by means of heuristic propagation rules that apply to each layer of a neural network [6]. In terms of computational cost the LRP method is very efficient, as it can be computed in one forward and backward pass through the network. In various applications LRP was shown to produce faithful explanations, even for highly complex and nonlinear networks used in computer vision [6, 64]. Besides it was able to detect biases in models and datasets used for training [44], e.g. the presence of a copyright tag that spuriously correlated to the class ‘horse’ in the Pascal VOC 2012 dataset. Further, it was used to get new insights in scientific and medical applications [71, 39, 77], to interpret clustering [40], to analyze audio data [75, 9], and to compare text classifiers for topic categorization [3].
Conservative Propagation.
LRP explains by redistributing the neural network output progressively from layer to layer until the input layer is reached. Similar to other works such as [41, 80, 67], the propagation procedure implemented by LRP is based on a local conservation principle: the net quantity, or relevance, received by any higher layer neuron is redistributed in the same amount to neurons of the layer below. In this way the relevance’s flow is analog to the Kirchhoff’s first law for the conservation of electric charge, or to the continuity equation in physics for transportation in general form. Concretely, if and are indices for neurons in two consecutive layers, and denoting by the relevance flowing between two neurons, we have the equations:
This local enforcement of conservation induces conservation at coarser scales, in particular, conservation between consecutive layers , and ultimately, conservation at the level of the whole deep neural network, i.e. given an input and its prediction , we have ^{1}^{1}1The global conservation is exact up to the relevance absorbed by some stabilizing term, and by the biases, see details later in Section 3.1.. This global conservation property allows to interpret the result as the share by which each input variable has contributed to the prediction.
LRP in Deep Neural Networks.
LRP has been most commonly applied to deep rectifier networks. In these networks, the activations at the current layer can be computed from activations in the previous layer as:
A general family of propagation rules for such types of layer is given by [51]:
Specific propagation rules such as LRP, LRP and LRP fall under this umbrella. They are easy to implement [42, 51] and can be interpreted as the result of a deep Taylor decomposition of the neural network function [52].
On convolutional neural networks for computer vision, composite strategies making use of different rules at different layers have shown to work well in practice [43, 51]. An alternative default strategy in computer vision is to uniformly employ the LRP in every hidden layer [53], the latter has the advantage of having no free parameter, and delivers positive explanations. On convolutional neural networks for text, LRP with a small value was found to work well [3, 57], it provides a signed explanation.
While LRP was described in the context of a layered feedforward neural network, the principle is general enough to apply to arbitrary directed acyclic graphs, including recurrent neural networks unfolded in time such as LSTMs.
3 Extending LRP for LSTMs
We address the question of how to explain the LSTM model’s output by expanding the previously described LRP technique to “standard” LSTM architectures, in the form they are most commonly used in the literature [26], i.e. following the recurrence Equations 16 and Figure 1 introduced in Section 2.1, and usually containing the tanh nonlinearity as an activation function for the cell input and the cell state.
For this, we first need to identify an appropriate structure of computation in these models, and introduce some notation. Let be the neurons representing the signal and the gate, let be the neuron representing the product of these two quantities. Let be the neuron corresponding to the forget gate. Let be the neuron on which the signal is being accumulated. Let be the same neurons at previous time steps. We can recompose the LSTM forward pass in terms of the following three elementary types of computation:
1. linear mappings  
2. gated interactions  
3. accumulation 
These three types of computation and the way they are typically interconnected are shown graphically in Figure 2.
Linear mappings form the input of the gated interactions. The output of some of the gated interactions enter into the accumulation function.
3.1 Linear Mappings
Each output of this computation is a weighted sum over a large number of input variables. Here, one strategy is to redistribute the relevance in proportion to the weighted activations , as they occur in the linear projection formulas above. One way of implementing this strategy is the epsilonrule (LRP) given by [6]:
where is a small stabilizer that pushes the denominator away from zero by some constant factor, and has the effect of absorbing some relevance when the weighted activations are weak or contradictory. This type of propagation rule was employed by previous works with recurrent neural networks [2, 4, 14, 57, 77]. A large value for tends to keep only the most salient factors of explanation. Note that, in our notation, neuron biases are taken into account via a constant neuron whose connection weight is the corresponding bias. This neuron also gets assigned a share of relevance. However its relevance will not be propagated further and will get trapped in that neuron, since the “bias neuron” has no lowerlayer connections.
3.2 Gated Interactions
These layers do not have a simple summing structure as the linear mappings. Their multiplicative nonlinearity makes them intrinsically more difficult to handle. Recently, three works extended the LRP propagation technique to recurrent neural networks, such as LSTMs [37] and GRUs [12], by proposing a rule to propagate the relevance through such product layers [2, 4, 14]. These LRP extensions were tested in the context of sentiment analysis, machine translation and reinforcement learning respectively. Arras et al. [4], in particular, proposed the signaltakeall redistribution rule
referred as “LRPall” in our experiments. This redistribution strategy can be motivated in a similar way the gates were initially introduced in the LSTM model [37]: the gate units are intended to control the flow of information in the LSTM, but not to be information themselves.
In the following, we provide further justification of this rule based on Deep Taylor Decomposition (DTD) [52], a mathematical framework for analyzing the relevance propagation process in a deep network. DTD expresses the relevance obtained at a given layer as a function of the activations in the lowerlayer, and determines how the relevance should be redistributed based on a Taylor expansion of this function. Consider the relevance function mapping the input of the gated interaction to the relevance received by the output of that module. We then write its Taylor expansion:
where is a root point of the function, and where the firstorder terms can be used to determine on which lowerlayer neurons ( or the relevance should be propagated). In practice, a root point and its gradient are difficult to compute analytically. However, we can consider instead a relevance model [52] which is easier to analyze, in our case, of the form:
The variable is constant and set such that locally. This model is a reasonable approximation when results from a propagation rule where the activation term naturally factors out (cf. [52]). The relevance model for the gated interaction of the standard LSTM is depicted in Figure 3 (left).
Having built the relevance model, we would like to perform a Taylor expansion of it at some root point in the vicinity of the observed point . The nearest root point of the relevance model is found at , and more generally any root point satisfies . A Taylor expansion of this simplified relevance model gives:
Clearly, the first linear term is zero for the nearest root point, thus, no relevance will be redistributed to the gate, however, the saturation effect of the hyperbolic tangent can create a mismatch between the firstorder term, and the function value to redistribute. However, if replacing in the LSTM the hyperbolic tangent by the identity or the ReLU nonlinearity (as this was done, for example, in [59]), then we get an exact decomposition of the relevance model with , since the Taylor remainder is exactly zero in this case. This corresponds to the LRPall redistribution rule.
This section has justified the signaltakeall strategy for standard LSTMs. In Section 4 modified LSTM variants that are tuned for further interpretability will benefit from a different propagation strategy. For example, using sigmoids both for the gate and the signal (cf. Figure 3 right) suggests a different propagation strategy.
A more complete set of propagation rules that have been used in practice [2, 4, 14, 57, 59, 77], and that we consider in our experiments, is given in Table 1. In addition to the definitions provided in Table 1, in order to avoid near zero division, one may add a stabilizing term into the denominator of the LRPprop and LRPabs variants, similarly to the epsilonrule stabilization for linear mappings. It has the form in the first case, and simply in the other case, where is a small positive number.
Name  Proposed in  Received by gate  Received by signal  DTD 
LRPall  [4]  
LRPprop  [14, 2]  
LRPabs  
LRPhalf  [2] 
3.3 Accumulation
The last type of module one needs to consider is the accumulation module that discounts the LSTM memory state with a “forget” factor, and adds a small additive term based on current observations:
Consider the relevance of the accumulator neuron for the final time step. Define . Through the accumulation module, we get the following redistribution:
where we have used the signaltakeall strategy in the product, and the epsilonrule (with no stabilizer) in the sum. Iterating this redistribution process on previous time steps, we obtain:
Note that, here, we assume a simplified recurrence structure over the standard LSTM presented in Figure 1, in particular we assume that neurons do not redistribute relevance to past time steps via (i.e. is connected only to the current input and not to previous recurrent states), to simplify the present analysis.
Now we inspect the structure of the relevance scores at each time step, as given above. We can see that the relevance terms can be divided into three parts:

A product of forget gates: This term tends to decrease exponentially with every further redistribution step, unless the forget gate is equal to one. In other words, only the few most recent time steps will receive relevance.

The value of the product neuron at the current time step. In other words, the relevance at a given time step is directly influenced by the activation of its representative neuron, which can be either positive or negative.

A term that does not change with the time steps, and relates to the amount of relevance available for redistribution.
These observations on the structure of the relevance over time provide a further justification for the LRP explanation procedure, which we will be validating empirically in Section 5. They also serve as a starting point to propose new variants of the LSTM for which the relevance redistribution satisfies further constraints, as proposed in the following Section 4.
4 LSTM Architectures Motivated by LRP
A “standard” LSTM network with fully connected LSTM blocks, as presented in Figure 1, is a very powerful network capable of modelling extremely complex sequential tasks. However, in many cases, an LSTM network with a reduced complexity is able to solve the same problems with a similar prediction performance. With the further goal of increasing the model’s interpretability, we propose some modifications which simplify the LSTM network and make the resulting model easier to explain with LRP.
4.1 LSTM for LRP Backward Analysis: Nondecreasing Memory Cells
The LRP backward propagation procedure is made simpler if memory cell states are nondecreasing, this way the contribution of each input to each memory cell is welldefined, and the problem that a negative and a positive contribution cancel each other is avoided. For nondecreasing memory cells and backward analysis with LRP, we make the following assumptions over the LSTM network from Figure 1 and Equations 16:

[label=(A0)]

for all . That is, the forget gate is always 1 and nothing is forgotten. This ensures uniform credit assignment over time, and alleviates the problem identified earlier in Section 3.3 that the relevance redistributed via the accumulation module decreases over time.

, that is, the cell input activation function is positive. For example we can use a sigmoid : , with . Indeed methods like LRP and the epsilonrule for linear mappings (cf. Section 3.1) face numerical stability issues when negative contributions cancel with positive contributions [53]. With a positive all contributions are positive, and the redistribution in the LSTM accumulation module is made more stable. Further, we assume that the cell input has a negative bias, that is, . This is important to avoid the drift effect. The drift effect is that the memory content only gets positive contributions, which leads to an increase of over time. Typical values are .

We want to ensure that for the cell state activation it holds , such that, if the memory content is zero, then nothing is transferred to the next layer. Therefore we set , with .

The cell input is only connected to the input, and is not connected to other LSTM memory cell outputs. Which means is zero. This ensures that LRP assigns relevance to the input and is not disturbed by redistributing relevance to the network.

The input gate has only connections to other memory cell outputs, and is not connected to the input. That is, is zero. This ensures that LRP assigns relevance only via to the input.

The output gate has only connections to other memory cell outputs, and is not connected to the input. That is, is zero. This ensures that LRP assigns relevance only via to the input.

The input gate has a negative bias, that is, . Like with the cell input the negative bias avoids the drift effect. Typical values are .

The output gate may also have a negative bias, that is, . This allows to bring in different memory cells at different time points. It is related to resource allocation.

The memory cell state content is initialized with zero at time , that is, . Lastly, the memory cell content is nonnegative for all , since and .
The resulting LSTM forward pass rules for LRP are:
cell input  (7)  
input gate  (8)  
cell state  (9)  
output gate  (10)  
cell output  (11) 
See Figure 4a which depicts these forward pass rules for LRP.
4.2 LSTM for LRP Backward Analysis: Keeping the Markov Property
Forget gates can modify the memory cells’ information at some future time step, i.e. they could completely erase the cells’ state content. Output gates can hide the cells’ information and deliver it in the future, i.e. output gates can be closed and open at some future time, masking all information stored by the cell. Thus, in order to guarantee the Markov property, the forget and output gates must be disconnected.
The resulting LSTM forward pass rules for Markov memory cells are:
cell input  (12)  
input gate  (13)  
cell state  (14)  
cell output  (15) 
See Figure 4b for an LSTM memory cell that guarantees the Markov property.
4.3 LSTM without Gates
The most simple LSTM architecture for backward analysis does not use any gates. Therefore complex dynamics that have to be treated in the LRP backward analysis are avoided.
The resulting LSTM forward pass rules are:
cell input  (16)  
cell state  (17)  
cell output  (18) 
Note that even this simple architecture can solve sequential problems, since different biases can be learned by the cell inputs to specialize on different time steps and activate the memory cell output accordingly.
See Figure 4c for an LSTM memory cell without gates which perfectly redistributes the relevance across the input sequence.
5 Experiments
5.1 Validating Explanations on Standard LSTMs: Selectivity and Fidelity
First we verify that the LRP explanation is able to select input positions that are the most determinant either in supporting or in contradicting an LSTM’s prediction, using a sentiment prediction task. To that end we perform a perturbation experiment aka “pixel flipping” or “region perturbation” [6, 64] commonly used in computer vision to evaluate and generate explanations, e.g. [47, 1, 11, 54]. Here we confirm whether the sign and ordering of the relevance reflect what the LSTM considers as highly speaking for or against a particular class.
Another property of the relevance we test is fidelity. To that end we use a synthetic task where the inputoutput relationship is known and compare the relevances w.r.t some ground truth explanation. By using a synthetic toy task we can avoid problems of disentangling errors made by the model from errors made by the explanation [72]. Here we seek to validate the magnitude of the relevance as a continuously distributed quantity. To the best of our knowledge we are the first one to conduct such a continuous analysis of the relevance in recurrent neural networks. Yet another work validated LSTM explanations via a toy classification task [77], however it practically treated the relevance as a binary variable.
Explanation Methods.
Now we introduce the various explanation methods we consider in our experiments with standard LSTMs. For the LRP explanation technique we consider all product rule variants specified in Table 1 (cf. Section 3.2) , i.e. LRPall [4], LRPprop [14, 2], LRPabs and LRPhalf [2]. Since the LRP backward pass delivers one relevance value per input dimension, we simply sum up the relevances across the input dimensions to get one relevance value per time step. Besides LRP we also consider gradientbased explanation [21, 45, 68, 13], occlusionbased relevance [46, 79], and Contextual Decomposition (CD) [56], as alternative methods.
For the gradientbased explanation we use as the relevance, either the prediction function’s partial derivative w.r.t. the input dimension of interest and square this quantity, we denote this variant simply as Gradient, or else we multiply this derivative by the input dimension’s value, we call it Gradient Input relevance. In both cases, similarly to LRP, the relevance of several input dimensions can be summed up to obtain one relevance value per time step.
For the occlusionbased explanation we take as the relevance, either a difference of prediction function values (i.e. of prediction scores before softmax normalization), we denote this variant as Occlusion, or else we use a difference of predicted probabilities, we call it Occlusion, where the difference is calculated over the model’s prediction on the original input and a prediction with an altered input where the position of interest (for which the relevance is being computed) is set to zero.
For the CD explanation method [56] we employ the code from the authors^{2}^{2}2https://github.com/jamiemurdoch/ContextualDecomposition to generate one relevance value per time step (this necessitates to run the CD decomposition as many times as there are time steps in the input sequence).
Testing Selectivity.
In order to assess the selectivity, we consider a fiveclass sentiment prediction task of movie reviews. As a dataset we use the Stanford Sentiment Treebank (SST) [69] which contains labels (from very negative, negative, neutral, positive, to very positive sentiment) for resp. 8544/1101/2210 train/val/test sentences and their constituent phrases. As an LSTM model we employ the bidirectional LSTM model from Li et al. [45] already trained on SST^{3}^{3}3https://github.com/jiweil/VisualizingandUnderstandingNeuralModelsinNLP and previously employed by the authors to perform a gradientbased analysis on the network decisions. The input consists of a sequence of 60dimensional word embeddings, the LSTM hidden layer has size 60, and the only text preprocessing is lowercasing. On binary sentiment classification of full sentences (ignoring the neutral class) the model reaches 82.9% test accuracy, and on fiveclass sentiment prediction of full sentences it achieves 46.3% accuracy.
For the perturbation experiment we consider all test sentences with a length of at least ten words (thus we retain 1849 sentences), and compute wordlevel relevances (i.e. one relevance value per time step) using as the target output class the true sentence class, and considering all five classes of the sentiment prediction task. For the computation of the LRP relevance we use as a stabilizer value, for linear mappings and product layers, ^{4}^{4}4Except for the LRPprop variant, where we take . We tried following values: [0.001, 0.01, 0.1, 0.2, 0.3, 0.4, 1.0], and took the lowest one to achieve numerical stability..
Then, given these wordlevel relevances, we iteratively remove up to five words from each input sentence, either in decreasing or increasing order of their relevance, depending on whether the corresponding sentence was initially correctly or falsely classified by the LSTM. We expect this input modification to decrease resp. increase the model’s confidence for the true class, which we measure in terms of the model’s accuracy. For removing a word we simply discard it from the input sequence and concatenate the remaining parts of the sentence. An alternative removal scheme would have been to set the corresponding word embedding to zero in the input (which in practice gave us similar results), however the former enables us to generate more natural texts, although we acknowledge that the resulting sentence might be partly syntactically broken as pointed out by Poerner et al. [57].
Our results of the perturbation experiment are compiled in Figure 5.
When looking at the removal of the most relevant words (Figure 5 left), we observe that the occlusionbased relevance, LRPall and CD are the most competitive methods, and perform onpar; followed by Gradient Input, which performs better than Gradient. The remaining methods, which are the other LRP variants LRPprop, LRPabs, LRPhalf are almost equivalent to random, and thus not adequate to detect words speaking for a specific class.
In the removal of the least relevant words (Figure 5 right), Occlusion performs best; followed by Occlusion, LRPall, CD and Gradient Input. Again the remaining LRP variants are almost equivalent to random. However this time Gradient performs worse than random, this indicates that low Gradient relevance is more likely to identify unimportant words for the classification problem (like stopwords) rather than identifying words speaking against a specific class (this was also observed in previous work, see e.g. [4] Table 1).
In summary, our perturbation experiment in sentiment analysis suggests that if one is interested in identifying the most influential positions that strongly support or inhibit a specific classification decision using a standard LSTM model, then the occlusionbased relevance, the LRP method with the product rule from Arras et al. [4], and the CD method of Murdoch et al. [56] are good candidates to provide this information.
For another evaluation of recurrent neural networks explanations, including a standard LSTM model, we further refer to Poerner et al. [57], in particular to their experiment using a subjectverb agreement task. Here the authors find that LRP and DeepLIFT [67] perform best among the tested explanation methods, both when using the signaltakeall strategy proposed in Arras et al. [4] for product layers^{5}^{5}5Ancona et al. [1] also performed a comparative study of explanations on LSTMs, however, in order to redistribute the relevance through product layers, the authors use standard gradient backpropagation. This redistribution scheme violates one of the key underlying property of LRP, which is local relevance conservation, hence their results for LRP are not conclusive..
Testing Fidelity.
In order to validate the fidelity, we consider a toy task with a linear inputoutput relationship. In particular we use the addition/subtraction of two numbers. Accordingly we expect the relevances to be linearly related to the actual input values, which we can directly measure in terms of the empirical correlation.
For our purpose we use a variant of the adding problem of Hochreiter et al. [36], where instead of using explicit markers, we use implicit ones; further we remove the sequence start and end positions. This way we enforce the LSTM model to attribute nonzero relevance only to the relevant numbers in the input and not to markers (since in general it is unclear what “ground truth” relevance should be attributed to a marker, to which we could compare the computed relevance to). Thus our input sequence of length has the form:
where the nonzero entries , with , are randomly sampled real numbers, and the two relevant positions and are sampled uniformly among with . The target output is for addition, and for subtraction. To ensure that the train/val/test sets do not overlap, we use 10000 sequences with for training, 2500 sequences with for validation, and 2500 sequences with as test set. Training is performed by minimizing Mean Squared Error (MSE).
More particularly, we consider two minimal tasks, which are solvable by a standard LSTM with only one memory cell, followed by a linear output layer with no bias (thus the model has 17 trainable parameters):

the addition of signed numbers (where is sampled uniformly from ),

the subtraction of positive numbers (where is sampled uniformly from )^{6}^{6}6We use an arbitrary minimum magnitude of 0.5 only to simplify training (since sampling very small numbers would encourage the model weights to grow rapidly)..
For each task we train 50 LSTM models with a validation MSE , the resulting test MSE is also .
Then, using the model’s predicted output, we compute one relevance value per time step , for each considered explanation method.
For the occlusionbased relevance we use only the Occlusion variant, since the model output is onedimensional and the considered task is a regression. For the gradientbased relevance we report only the Gradient Input results, since the pure Gradient performs very poorly. For the computation of the LRP relevance we didn’t find it necessary to add any stabilizing term (therefore we use for all LRP rules).
Our results are reported in Table 2. For the positions and , we checked whether there is a correlation between the computed relevance and the input numbers’ actual value. Besides, we verified the portion of the relevance (in absolute value) assigned to these positions, compared to the relevance attributed to all time steps in the sequence.
Addition  
Gradient Input  99.960 (0.017)  99.954 (0.019)  99.68 (0.53) 
Occlusion  99.990 (0.004)  99.990 (0.004)  99.82 (0.27) 
LRPprop  0.785 (3.619)  10.111 (12.362)  18.14 (4.23) 
LRPabs  7.002 (6.224)  12.410 (17.440)  18.01 (4.48) 
LRPhalf  29.035 (9.478)  51.460 (19.939)  54.09 (17.53) 
LRPall  99.995 (0.002)  99.995 (0.002)  99.95 (0.05) 
CD  99.997 (0.002)  99.997 (0.002)  99.92 (0.06) 
Subtraction  
Gradient Input  97.9 (1.6)  98.8 (0.6)  98.3 (0.6) 
Occlusion  99.0 (2.0)  69.0 (19.1)  25.4 (16.8) 
LRPprop  3.1 (4.8)  8.4 (18.9)  15.0 (2.4) 
LRPabs  1.2 (7.6)  23.0 (11.1)  15.1 (1.6) 
LRPhalf  7.7 (15.3)  28.9 (6.4)  42.3 (8.3) 
LRPall  98.5 (3.5)  99.3 (1.3)  99.3 (0.6) 
CD  25.9 (39.1)  50.0 (29.2)  49.4 (26.1) 
Interestingly several methods pass our sanity check (they are reported in bold in the Table) and attribute as expected a correlation of almost one in the addition task, namely: Gradient Input, Occlusion, LRPall and CD.
However, on subtraction, only Gradient Input and LRPall assign a correct correlation of near one to the first number, and of near minus one to the second number, while the remaining explanation methods fail completely.
For both addition and subtraction, we observe that methods that fail in the correlation results also erroneously assign a nonnegligible portion of the relevance to clearly unimportant time steps.
One key difference between the two arithmetic tasks we considered, is that our addition task is nonsequential and solvable by a BagofWords approach (i.e. by ignoring the ordering of the inputs), while our subtraction task is truly sequential and requires the LSTM model to remember which number arrives in the first position and which number in the second.
From this sanity check, we certainly can not deduce that every method that passes the subtraction test is also appropriate to explain any complex nonlinear prediction task. However, we postulate that an explanation method that fails on such a test with the smallest possible number of free parameters (i.e. an LSTM with one memory cell) is generally a less suited method.
In this vein, our present analysis opens up new avenues for improving and testing LSTM explanation methods in general, including the LRP method and its LRPall variant, whose results in our arithmetic task degrade when more memory cells are included to the LSTM model, which suggests that gates might also be used by the standard LSTM to store the input numbers’ value^{7}^{7}7The same phenomenon can occur, on the addition problem, when using only positive numbers as input. Whereas in the specific toy tasks we considered, the cell input () is required to process the numbers to add/subtract, and the cell state () accumulates the result of the arithmetic operation.. The latter phenomenon could be either avoided by adapting the LSTM architecture, or could be taken into account in the relevance propagation procedure by employing alternative propagation rules for products. This leads us to the next subsection, where we use an adapted LSTM model and different product rules, for the task of reward redistribution.
5.2 Long Term Credit Assignment in Markov Decision Processes via LRP and LSTMs
Assigning the credit for a received reward to actions that were performed is one of the central tasks in reinforcement learning [74]. Long term credit assignment has been identified as one of the biggest challenges in reinforcement learning [62]. Classical reinforcement learning methods use a forward view approach by estimating the future expected return of a Markov Decision Process (MDP). However, they fail when the reward is delayed since they have to average over a large number of probabilistic future stateaction paths that increases exponentially with the delay of the reward [58, 48].
In contrast to using a forward view, a backward view approach based on a backward analysis of a forward model avoids problems with unknown future stateaction paths, since the sequence is already completed and known. Backward analysis transforms the forward view approach into a regression task, at which deep learning methods excel. As a forward model, an LSTM can be trained to predict the final return, given a sequence of stateactions. LSTM was already used in reinforcement learning [66] for advantage learning [7] and learning policies [27, 50, 28]. However, backward analysis via sensitivity analysis like “backpropagation through a model” [55, 60, 61, 8] have major drawbacks: local minima, instabilities, exploding or vanishing gradients in the world model, proper exploration, actions being only regarded by sensitivity but not their contribution (relevance) [29, 65].
Contribution analysis, however, can be used to decompose the return prediction (the output relevance) into contributions of single stateaction pairs along the observed sequence, obtaining a redistributed reward (the relevance redistribution). As a result, a new MDP is created with the same optimal policies and, in the optimal case, with no delayed rewards (expected future rewards equal zero) [2]. Indeed, for MDPs the value is equal to the expected immediate reward plus the expected future rewards. Thus, if the expected future rewards are zero, the value estimation simplifies to computing the mean of the immediate rewards.
In the following experiment we do not evaluate the performance of the agent under this reward redistribution. Instead, the aim of this experiment is to show how different LRP product rules change the explanation of the model and, therefore, the reward redistribution.
LSTM and Markov Properties.
For LSTMs with forget gate or output gate [26], as described in Section 2.1, the cell content does not comply to Markov assumptions. This is because the current contribution of an input to a memory cell can be modified or hidden by later inputs, via the forget gate or output gate. For example, the forget gate can erase or reduce the contribution of the current input in the future due to some later inputs. Likewise, the output gate can hide the contribution of the current input by closing the gate until some future input opens the gate and reveals the already past contribution.
Figure 4b shows an LSTM memory cell that complies with the Markov property.
Environment.
In our environment, an agent has to collect the Moneybag and then collect as many Coins as possible in a one dimension grid world. Only Coins collected after collecting the Moneybag give reward. At each time step, the agent can move to the left or to the right. All rewards are only given at the end of the episode, depending on how many Coins the agent collected in the Moneybag.
Training the Model.
We are given a collection of sequences of stateaction pairs. Each sequence represents one episode. Each episode is labeled with a scalar corresponding to the episode return. States and actions in the sequence are encoded as a vector of four binary features representing if the Moneybag is collected, if a Coin is collected, and the chosen action for the current timestep (one binary feature per action). In this experiment, we use a Long ShortTerm Memory (LSTM) [30, 37] to predict the return of an episode [2]. Notice that since we are using an LSTM, the input encoding does not have to fulfil the Markov property. Once the LSTM is trained, we use LRP [6] as contribution analysis for backward analysis.
LRP and Different Product Rules.
We trained an LSTM network, as depicted in Figure 4b, to predict the return given a sequence of states and actions. Later, we applied different product rules to propagate the relevance through the input gates and the cell inputs (cf. Table 1 Section 3.2). Results are shown in Figure 6 and 7. When no relevance is propagated through the input gates (LRPall), certain important events are not detected by the contribution analysis, i.e. no relevance is assigned to the event Moneybag. This is due to the representation learned by the LSTM model, which stores information about the Moneybag feature, which in turn is used to activate a learned Coins counter via the input gate once the Moneybag has been collected. As such, the Moneybag event contributes through the input gate. When relevance is allowed to flow through the input gates (LRPprop and LRPhalf), all events can be detected, including the actions that lead to the Moneybag event. However, the amount of relevance is not completely conserved when it is propagated through the gates, as a small amount of relevance can get trapped in the LSTM cell, in particular via the relevance of the initial time step input gate.
6 Conclusion
We presented several ways of extending the LRP technique to recurrent neural networks such as LSTMs, which encompasses defining a rule to propagate the relevance through product layers. Among the tested product rule variants we showed that, on standard LSTM models, the signaltakeall strategy leads to the most pertinent results, and can be embedded in the deep Taylor framework where it corresponds to choosing the nearest root point in the gated interaction relevance model.
Additionally, we showed that the relevance propagation flow can be made more straightforward and stable by adapting the LSTM model towards the LRP technique and that, in this case, propagating a share of relevance through the gates leads to a detection of relevant events earlier in time. The resulting simplified and less connected LSTM model can potentially solve the same problems as the standard LSTM, although in practice this may necessitate using more memory cells.
More generally, further investigating the representational power of the new proposed LSTM, as well as its interplay with various LRP propagation rules, in particular via controlled experiments, would be a subject for future work.
Acknowledgements. This work was supported by the German Ministry for Education and Research as Berlin Big Data Centre (01IS14013A), Berlin Center for Machine Learning (01IS18037I) and TraMeExCo (01IS18056A). Partial funding by DFG is acknowledged (EXC 2046/1, projectID: 390685689). This work was also supported by the Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (No. 2017000451, No. 2017001779).
References
 [1] Ancona, M., Ceolini, E., Öztireli, C., Gross, M.: Towards better understanding of gradientbased attribution methods for deep neural networks. In: International Conference on Learning Representations (ICLR) (2018)
 [2] ArjonaMedina, J.A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., Hochreiter, S.: RUDDER: Return decomposition for delayed rewards. To appear in: Advances in Neural Information Processing Systems 32 (NeurIPS 2019) arXiv:1806.07857 (2018)
 [3] Arras, L., Horn, F., Montavon, G., Müller, K.R., Samek, W.: “What is relevant in a text document?”: An interpretable machine learning approach. PLoS ONE 12(8), e0181142 (2017)
 [4] Arras, L., Montavon, G., Müller, K.R., Samek, W.: Explaining recurrent neural network predictions in sentiment analysis. In: Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), pp. 159–168. Association for Computational Linguistics (2017)
 [5] Arras, L., Osman, A., Müller, K.R., Samek, W.: Evaluating recurrent neural network explanations. In: Proceedings of the ACL 2019 Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 113–126. Association for Computational Linguistics (2019)
 [6] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixelwise explanations for nonlinear classifier decisions by layerwise relevance propagation. PLoS ONE 10(7), e0130140 (2015)
 [7] Bakker, B.: Reinforcement learning with long shortterm memory. In: Advances in Neural Information Processing Systems 14 (NIPS). pp. 1475–1482 (2002)
 [8] Bakker, B.: Reinforcement learning by backpropagation through an LSTM model/critic. In: IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning. pp. 127–134 (2007)
 [9] Becker, S., Ackermann, M., Lapuschkin, S., Müller, K.R., Samek, W.: Interpreting and explaining deep neural networks for classification of audio signals. arXiv:1807.03418 (2018)
 [10] Bengio, Y., Simard, P., Frasconi, P.: Learning longterm dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (1994)
 [11] Chen, J., Song, L., Wainwright, M., Jordan, M.: Learning to explain: An informationtheoretic perspective on model interpretation. In: Proceedings of the 35th International Conference on Machine Learning (ICML). vol. 80, pp. 883–892 (2018)
 [12] Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoderdecoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1724–1734. Association for Computational Linguistics (2014)
 [13] Denil, M., Demiraj, A., de Freitas, N.: Extraction of salient sentences from labelled documents. arXiv:1412.6815 (2015)
 [14] Ding, Y., Liu, Y., Luan, H., Sun, M.: Visualizing and understanding neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 1150–1159. Association for Computational Linguistics (2017)
 [15] Donahue, J., Hendricks, L.A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., Darrell, T.: Longterm recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(4), 677–691 (2017)
 [16] EUGDPR: Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union L 119 59, 1–88 (2016)
 [17] Geiger, J.T., Zhang, Z., Weninger, F., Schuller, B., Rigoll, G.: Robust speech recognition using long shortterm memory recurrent neural networks for hybrid acoustic modelling. In: Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH). pp. 631–635 (2014)
 [18] Gers, F.A., Schmidhuber, J.: Recurrent nets that time and count. In: Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN). vol. 3, pp. 189–194 (2000)
 [19] Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with LSTM. In: Proceedings of the International Conference on Artificial Neural Networks (ICANN). vol. 2, pp. 850–855 (1999)
 [20] Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with LSTM. Neural Computation 12(10), 2451–2471 (2000)
 [21] Gevrey, M., Dimopoulos, I., Lek, S.: Review and comparison of methods to study the contribution of variables in artificial neural network models. Ecological Modelling 160(3), 249–264 (2003)
 [22] GonzalezDominguez, J., LopezMoreno, I., Sak, H., GonzalezRodriguez, J., Moreno, P.J.: Automatic language identification using long shortterm memory recurrent neural networks. In: Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH). pp. 2155–2159 (2014)
 [23] Graves, A.: Generating sequences with recurrent neural networks. arXiv:1308.0850 (2014)
 [24] Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(5), 855–868 (2009)
 [25] Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18(56), 602–610 (2005)
 [26] Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28(10), 2222–2232 (2017)
 [27] Hausknecht, M., Stone, P.: Deep recurrent qlearning for partially observable MDPs. In: AAAI Fall Symposium Series  Sequential Decision Making for Intelligent Agents. pp. 29–37 (2015)
 [28] Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., Silver, D.: Learning and transfer of modulated locomotor controllers. arXiv:1610.05182 (2016)
 [29] Hochreiter, S.: Implementierung und Anwendung eines ‘neuronalen’ EchtzeitLernalgorithmus für reaktive Umgebungen. Practical work, Institut für Informatik, Technische Universität München (1990)
 [30] Hochreiter, S.: Untersuchungen zu dynamischen neuronalen Netzen. Master’s thesis. Institut für Informatik, Technische Universität München (1991)
 [31] Hochreiter, S.: Recurrent neural net learning and vanishing gradient. In: Freksa, C. (ed.) Proceedings in Artificial Intelligence  FuzzyNeuroSysteme’97 Workshop. pp. 130–137. Infix (1997)
 [32] Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and KnowledgeBased Systems 6(2), 107–116 (1998)
 [33] Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning longterm dependencies. In: Kolen, J.F., Kremer, S.C. (eds.) A Field Guide to Dynamical Recurrent Networks, pp. 237–244. IEEE Press (2001)
 [34] Hochreiter, S., Heusel, M., Obermayer, K.: Fast modelbased protein homology detection without alignment. Bioinformatics 23(14), 1728–1736 (2007)
 [35] Hochreiter, S., Schmidhuber, J.: Long shortterm memory. Tech. Rep. FKI20795, Fakultät für Informatik, Technische Universität München (1995)
 [36] Hochreiter, S., Schmidhuber, J.: LSTM can solve hard long time lag problems. In: Advances in Neural Information Processing Systems 9 (NIPS). pp. 473–479 (1996)
 [37] Hochreiter, S., Schmidhuber, J.: Long shortterm memory. Neural Computation 9(8), 1735–1780 (1997)
 [38] Hochreiter, S., Younger, A.S., Conwell, P.R.: Learning to learn using gradient descent. In: Proceedings of the International Conference on Artificial Neural Networks (ICANN). pp. 87–94 (2001)
 [39] Horst, F., Lapuschkin, S., Samek, W., Müller, K.R., Schöllhorn, W.I.: Explaining the unique nature of individual gait patterns with deep learning. Scientific Reports 9, 2391 (2019)
 [40] Kauffmann, J., Esders, M., Montavon, G., Samek, W., Müller, K.R.,: From clustering to cluster explanations via neural networks. arXiv:1906.07633 (2019)
 [41] Landecker, W., Thomure, M.D., Bettencourt, L.M.A., Mitchell, M., Kenyon, G.T., Brumby, S.P.: Interpreting individual classifications of hierarchical networks. In: IEEE Symposium on Computational Intelligence and Data Mining (CIDM). pp. 32–38 (2013)
 [42] Lapuschkin, S., Binder, A., Montavon, G., Müller, K.R., Samek, W.: The LRP toolbox for artificial neural networks. Journal of Machine Learning Research 17(114), 1–5 (2016)
 [43] Lapuschkin, S., Binder, A., Müller, K.R., Samek, W.: Understanding and comparing deep neural networks for age and gender classification. In: IEEE International Conference on Computer Vision Workshops. pp. 1629–1638 (2017)
 [44] Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever hans predictors and assessing what machines really learn. Nature Communications 10, 1096 (2019)
 [45] Li, J., Chen, X., Hovy, E., Jurafsky, D.: Visualizing and understanding neural models in NLP. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACLHLT). pp. 681–691. Association for Computational Linguistics (2016)
 [46] Li, J., Monroe, W., Jurafsky, D.: Understanding neural networks through representation erasure. arXiv:1612.08220 (2017)
 [47] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems 30 (NIPS). pp. 4765–4774 (2017)
 [48] Luoma, J., Ruutu, S., King, A.W., Tikkanen, H.: Time delays, competitive interdependence, and firm performance. Strategic Management Journal 38(3), 506–525 (2017)
 [49] Marchi, E., Ferroni, G., Eyben, F., Gabrielli, L., Squartini, S., Schuller, B.: Multiresolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 2164–2168 (2014)
 [50] Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning (ICML). vol. 48, pp. 1928–1937 (2016)
 [51] Montavon, G., Binder, A., Lapuschkin, S., Samek, W., Müller, K.R.: Layerwise relevance propagation: An overview. In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science 11700, Springer (2019)
 [52] Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition 65, 211–222 (2017)
 [53] Montavon, G., Samek, W., Müller, K.R.: Methods for interpreting and understanding deep neural networks. Digital Signal Processing 73, 1–15 (2018)
 [54] Morcos, A.S., Barrett, D.G., Rabinowitz, N.C., Botvinick, M.: On the importance of single directions for generalization. In: International Conference on Learning Representations (ICLR) (2018)
 [55] Munro, P.: A dual backpropagation scheme for scalar reward learning. In: Proceedings of the Ninth Annual Conference of the Cognitive Science Society. pp. 165–176 (1987)
 [56] Murdoch, W.J., Liu, P.J., Yu, B.: Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In: International Conference on Learning Representations (ICLR) (2018)
 [57] Poerner, N., Schütze, H., Roth, B.: Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 340–350. Association for Computational Linguistics (2018)
 [58] Rahmandad, H., Repenning, N., Sterman, J.: Effects of feedback delay on learning. System Dynamics Review 25(4), 309–338 (2009)
 [59] Rieger, L., Chormai, P., Montavon, G., Hansen, L.K., Müller, K.R.: Structuring neural networks for more explainable predictions. In: Escalante H. et al. (eds.) Explainable and Interpretable Models in Computer Vision and Machine Learning. TSSCML, pp. 115–131. Springer, Cham (2018)
 [60] Robinson, A.J.: Dynamic error propagation networks. Ph.D. thesis, Trinity Hall and Cambridge University Engineering Department (1989)
 [61] Robinson, T., Fallside, F.: Dynamic reinforcement driven error propagation networks with application to game playing. In: Proceedings of the 11th Conference of the Cognitive Science Society, Ann Arbor. pp. 836–843 (1989)
 [62] Sahni, H.: Reinforcement learning never worked, and ’deep’ only helped a bit. himanshusahni.github.io/2018/02/23/reinforcementlearningneverworked.html (2018)
 [63] Sak, H., Senior, A., Beaufays, F.: Long shortterm memory recurrent neural network architectures for large scale acoustic modeling. In: Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH). pp. 338–342. Singapore (2014)
 [64] Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.R.: Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems 28(11), 2660–2673 (2017)
 [65] Schmidhuber, J.: Making the world differentiable: On using fully recurrent selfsupervised neural networks for dynamic reinforcement learning and planning in nonstationary environments. Tech. Rep. FKI12690 (revised), Institut für Informatik, Technische Universität München (1990), Experiments by Sepp Hochreiter
 [66] Schmidhuber, J.: Deep learning in neural networks: An overview. Neural Networks 61, 85–117 (2015)
 [67] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: Proceedings of the 34th International Conference on Machine Learning (ICML). vol. 70, pp. 3145–3153 (2017)
 [68] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. In: International Conference on Learning Representations (ICLR) (2014)
 [69] Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1631–1642. Association for Computational Linguistics (2013)
 [70] Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using LSTMs. In: Proceedings of the 32nd International Conference on Machine Learning (ICML). vol. 37, pp. 843–852 (2015)
 [71] Sturm, I., Lapuschkin, S., Samek, W., Müller, K.R.: Interpretable deep neural networks for singletrial EEG classification. Journal of Neuroscience Methods 274, 141–145 (2016)
 [72] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML). vol. 70, pp. 3319–3328 (2017)
 [73] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems 27 (NIPS). pp. 3104–3112 (2014)
 [74] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT Press, Cambridge, MA, 2 edn. (2017), draft from November 2017
 [75] Thuillier, E., Gamper, H., Tashev, I.J.: Spatial audio feature discovery with convolutional neural networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6797–6801 (2018)
 [76] Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.: Translating videos to natural language using deep recurrent neural networks. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACLHLT). pp. 1494–1504. Association for Computational Linguistics (2015)
 [77] Yang, Y., Tresp, V., Wunderle, M., Fasching, P.A.: Explaining therapy predictions with layerwise relevance propagation in neural networks. In: IEEE International Conference on Healthcare Informatics (ICHI). pp. 152–162 (2018)
 [78] Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. arXiv:1409.2329 (2015)
 [79] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Proceedings of the European Conference on Computer Vision (ECCV). LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014)
 [80] Zhang, J., Lin, Z., Brandt, J., Shen, X., Sclaroff, S.: Topdown neural attention by excitation backprop. In: Proceedings of the European Conference on Computer Vision (ECCV). LNCS, vol. 9908, pp. 543–559. Springer, Cham (2016)