Explaining Explanations
Abstract
Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model’s prediction on a given input. However, for many tasks, simply knowing which features were important to a model’s prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others.
In this work, we present Integrated Hessians
1 Introduction and Prior Work
Deep neural networks are among the most popular class of machine learning model. They have achieved stateoftheart performance in problem domains ranging from natural language processing Devlin et al. (2018) to image recognition He et al. (2016). They have even outperformed other nonlinear model types on structured tabular data Shavitt and Segal (2018). Although neural networks have been traditionally difficult to interpret compared to simpler model classes, gaining a better understanding of their predictions is desirable for a variety of reasons. To the extent that these algorithms are used in automated decisions impacting humans, explanations may be legally required Selbst and Powles (2017). When used in high stakes applications, like diagnostic radiology, it is essential to ensure that models are making safe decisions for the right reasons Geis et al. (2019). During model development, interpretability methods may generally help debug undesirable model behavior Sundararajan et al. (2017).
Feature attribution: There has been a large number of recent approaches to interpret deep neural networks, ranging from methods that aim to distill complex models into more simple models Tan et al. (2018); Wu et al. (2018); Puri et al. (2017), to methods that aim to identify the most important concepts learned by a network Kim et al. (2017); Olah et al. (2018, 2017); Fong and Vedaldi (2018); Erhan et al. (2009); Mahendran and Vedaldi (2015). One of the beststudied sets of approaches is known as feature attribution methods Binder et al. (2016); Shrikumar et al. (2017); Lundberg and Lee (2017); Ribeiro et al. (2016). These approaches explain a model’s prediction by assigning credit to each input feature based on how much it influenced that prediction. Although these approaches help practitioners understand which features are important, they do not explain why certain features are important or how features interact in a model. In order to develop a richer understanding of model behavior, it is therefore desirable to develop methods to explain not only feature attributions but also feature interactions. For example, in Figure 1, we show that wordlevel interactions can help us distinguish why deeper, more expressive neural networks outperform simpler ones on language tasks.
Feature interaction: There are several existing methods that explain feature interactions in neural networks. For example, Cui et al. (2019) propose a method to explain global interactions in Bayesian Neural Networks (BNN). After jointly training a linear model to learn main effects with a BNN to learn interactions, they detect learned interactions by finding pairs of features that on average (across many samples) have large magnitude terms in the input hessian of the BNN. Tsang et al. (2017) propose a framework called Neural Interaction Detection, that detects statistical interactions between features by examining the weight matrices of feedforward neural networks. In the domain of deep learning for genomics, Greenside et al. (2018) propose an approach called Deep Feature Interaction Maps. This approach detects interactions between a source and target feature by calculating the change in the firstorder feature attribution of the source feature when all other features are held constant but the target feature is changed.
Limitations of Prior Approaches: Recent work has shown that attempting to quantitatively or qualitatively compare methods to explain machine learning models can lead to misleading and unreliable results Tomsett et al. (2019); Buçinca et al. (2020). Instead, we attempt to draw contrast with previous approaches from a theoretical and practical perspective.
While previous approaches have taken important steps towards understanding feature interaction in neural networks, they all suffer from practical limitations. Each of the above methods are only applicable to models trained on certain types of data, or applicable only to certain model architectures. Neural Interaction Detection only applies to feedforward neural network architectures, and can not be used on networks with convolutions, recurrent units, or selfattention. The approach suggested by Cui et al. (2019) is limited in that it requires the use of Bayesian Neural Networks; it is unclear how to apply the method to standard neural networks. Deep Feature Interaction Maps only work when the input features for a model have a small number of discrete values (such as the genomic data used as input in their paper).
In addition to architecture and data limitations, existing methods to detect interactions do not satisfy the commonsense axioms that have been proposed by feature attribution methods Sundararajan et al. (2017); Lundberg and Lee (2017). This failure to ground interactions in existing theory leads these previous approaches being provably unable to find interactions when interactions are present, or more generally finding counterintuitive interactions (see section 4).
Our Contributions: (1) We propose an approach to quantifying pairwise feature interactions that can be applied to any neural network architecture; (2) We identify several commonsense axioms that featurelevel interactions should satisfy and show that our proposed method satisfies them; (3) We provide a principled way to compute interactions in ReLUbased networks, which are piecewise linear and have zero second derivatives; (4) We demonstrate the utility of our method by showing the insights it reveals into real productiontype models and real datasets.
2 Explaining Explanations with Integrated Hessians
To derive our feature interaction values, we start by considering Integrated Gradients, proposed by Sundararajan et al. (2017). We represent our model as a function .
(1) 
where is the sample we would like to explain and is some uninformative baseline value. Although is often a neural network, the only requirement in order to compute attribution values is that be differentiable along the path from to . Our key insight is that the Integrated Gradients value for a differentiable model is itself a differentiable function . This means that we can apply Integrated Gradients to itself in order to explain how much feature impacted the importance of feature :
(2) 
For , we can derive that:
(3) 
In the case of , the formula has an additional firstorder term. We leave the derivation and the full formula to the appendix. In this view, we can interpret as the explanation of the importance of feature in terms of the input value of feature .
2.1 Fundamental Axioms for Interaction Values
Sundararajan et al. (2017) showed that, among other theoretical properties, Integrated Gradients satisfies the completeness axiom, which states:
(4) 
Although we leave the derivation to the appendix, we can show the following two equalities, which are immediate consequences of completeness:
(5)  
(6) 
We call equation 5 the interaction completeness axiom: the sum of the terms adds up to the difference between the output of at and at the baseline . This axiom lends itself to another natural interpretation of : as the interaction between features and . That is, it represents the contribution that the pair of features and together add to the output . Satisfying interaction completeness is important because it demonstrates a relationship between model output and interaction values. Without this axiom, it is unclear how to interpret the scale of interactions.
Equation 6 provides a way to interpret the selfinteraction term : it is the main effect of feature after interactions with all other features have been subtracted away. We note that equation 6 also implies the following, intuitive property about the main effect: if for all , or in the degenerate case where is the only feature, we have . We call this the self completeness axiom. Satisfying selfcompleteness is important because it provides a guarantee that the main effect of feature equals its feature attribution value if that feature interacts with no other features.
Integrated Gradients satisfies several other commonsense axioms such as sensitivity and linearity. We discuss generalizations of these axioms to interaction values (interaction sensitivity and interaction linearity) and demonstrate the Integrated Hessians satisfies said axioms in the appendix. We do observe one additional property here, which we call interaction symmetry: that for any , we have . It is straightforward to show that existing neural networks and their activation functions have continuous second partial derivatives, which implies that Integrated Hessians satisfies interaction symmetry.
2.2 On Computing Integrated Hessians
In practice we compute the Integrated Hessian values by a discrete sum approximation of the integral, similar to how Integrated Gradients is computed Sundararajan et al. (2017). The original paper uses the completeness axiom to determine convergence of attribution values. Similarly, we can use the interaction completeness axiom to determine convergence of the discrete sum approximation. We find that anywhere between 50 to 300 discrete steps suffice to approximate the double integral in most cases. We leave the exact approximation formulas and a further discussion of convergence to the appendix, as well as section 3.
2.3 Baselines and Expected Hessians
Several existing feature attribution methods have pointed out the need to explain relative to a baseline value that represents a lack of information that the explanations are relative to Sundararajan et al. (2017); Shrikumar et al. (2017); Binder et al. (2016); Lundberg and Lee (2017). However, more recent work has pointed out that choosing a single baseline value to represent lack of information can be challenging in certain domains Kindermans et al. (2019); Kapishnikov et al. (2019); Sundararajan and Taly (2018); Ancona et al. (2017); Fong and Vedaldi (2017); Sturmfels et al. (2020). As an alternative, Erion et al. (2019) proposed an extension of Integrated Gradients called Expected Gradients, which samples many baseline inputs from the training set. It is defined as:
(7) 
where the expectation is over both for the training distribution and . We can apply Expected Gradients to itself to get Expected Hessians:
(8) 
where the expectation is over , and . In our experiments, we use Integrated Hessians where there is a natural baseline: for language models (Section 5.1) we use the zeroembedding as a baseline, and for gene expression data (Section 5.3) we use the zero vector. On our tabular data examples (Sections 5.2 and 5.4) we use Expected Hessians. We further discuss the hyperparameters used to compute interactions in the appendix.
3 Smoothing ReLU Networks
One major limitation that has not been discussed in previous approaches to interaction detection in neural networks is related to the fact that many popular neural network architectures use the ReLU activation function, . Neural networks that use ReLU are piecewise linear and have second partial derivatives equal to zero in all places. Previous secondorder approaches fail to detect any interaction in ReLUbased networks.
Fortunately, the ReLU activation function has a smooth approximation – the SoftPlus function:
(9) 
SoftPlus more closely approximates the ReLU function as increases, and has welldefined higherorder derivatives. Furthermore, Dombrowski et al. (2019) have proved that both the outputs and firstorder feature attributions of a model are minimally perturbed when ReLU activations are replaced by SoftPlus activations in a trained network. Therefore, one important insight of our approach is that Integrated Hessians for a trained ReLU network can be obtained by first replacing the ReLU activations with SoftPlus activations, then calculating the interactions. We note that no retraining is necessary for this approach.
In addition to being twice differentiable and allowing us to calculate interaction values in ReLU networks, replacing ReLU with SoftPlus leads to other desirable behavior for calculating interaction values. We show that smoothing a neural network (decreasing the value of in the SoftPlus activation function) lets us accurately approximate the Integrated Hessians value with fewer gradient calls.
Theorem 1.
For a onelayer neural network with softplus\textsubscript nonlinearity, softplus\textsubscript, and input features, we can bound the number of interpolation points needed to approximate the Integrated Hessians to a given error tolerance by .
Proof of Theorem 1 is contained in the appendix. In addition to the proof for the singlelayer case, in the appendix we also show empirical results that manylayered neural networks display the same property. Finally, we demonstrate the intuition behind these results. As we replace ReLU with SoftPlus, the decision surface of the network is smoothed  see Figure 2 and Dombrowski et al. (2019). We can see that the gradients tend to all have more similar direction along the path from reference to foreground sample once the network has been smoothed with SoftPlus replacement.
4 Explanation of XOR function
To understand why feature interactions can be more informative than feature attributions, we can first consider the explanations for the XOR function. This illustration also highlights a weakness of any method for detecting interactions that uses the Hessian without integrating over a path. Our data consists of two binary features, and the function learned by our model is the exclusive OR function – it has a large magnitude output when either one feature or the other is on, but a low magnitude output when both features are either on or off (see Figure 3, left).
We use the point (0,0) as a baseline. The Integrated Gradients feature attributions for the states where both features are off (0,0) and both features are on (1,1) are identical: both features get 0 attribution because (see Figure 3, right upper). When we use Integrated Hessians to get all of the pairwise interactions between features for the two samples, we see that the samples now have different explanations in terms of interactions (see Figure 3, right lower). The first datapoint (0,0), which is identical to the baseline, has all 0 interaction values. The second point (1,1), now has negative interaction between the two features and positive selfinteraction. Therefore the interactions are usefully able to distinguish between (0,0), which has an output of 0 because it is identical to the baseline, and (1,1), which has an output of 0 because both features are on, which on their own should increase the model output, but in interaction with each other cancel out the positive effects and drive the model’s output back to the baseline.
This example also illustrates a problem with methods like Cui et al. (2019), which identifies global interactions between features and by measuring the average magnitude of the th element of the Hessian over all samples. Looking at Figure 3, we can see that the input gradients and input hessians have completely flattened (saturated) at all points on the data manifold, and consequently the magnitude of the th element of the Hessian over all samples will be 0. By integrating between the baseline and the samples, Integrated Hessians is capable of correctly detecting the negative interaction between the two features.
5 Applications of Integrated Hessians
In this section, we outline four different use cases of Integrated Hessians on realworld data in order to demonstrate its broad applicability.
5.1 Nlp
In the past decade, neural networks have been the goto model for language tasks, from convolutional Kim (2014) to recurrent Sundermeyer et al. (2012). More recently, attention mechanisms Vaswani et al. (2017) and large, pretrained transformer architectures Peters et al. (2018); Devlin et al. (2018) have achieved state of the art performance on a wide variety of tasks.
Some previous work has suggested looking at the internal weights of the attention mechanisms in models that use attention Ghaeini et al. (2018); Lee et al. (2017); Lin et al. (2017); Wang et al. (2016). However, more recent work has suggested that looking at attention weights may not be a reliable way to interpret models with attention layers Serrano and Smith (2019); Jain and Wallace (2019); Brunner et al. (2019). To overcome this, feature attributions have been applied to text classification models to understand which words most impacted the classification Liu and Avci (2019); Lai et al. (2019). However, these methods do not explain how words interact with their surrounding context.
We download pretrained weights for DistilBERT Sanh et al. (2019) from the HuggingFace Transformers library Wolf et al. (2019). We finetune the model on the Stanford Sentiment Treebank dataset Socher et al. (2013) in which the task is to predict whether or not a movie review has positive or negative sentiment. After 3 epochs of finetuning, DistilBERT achieves a validation accuracy of 0.9071 (0.9054 TPR / 0.9089 TNR).
In Figure 4, we show interactions generated by Integrated Hessians and attributions generated by Integrated Gradients on an example drawn from the validation set. The figure demonstrates that DistilBERT has learned intuitive interactions that would not be revealed from feature attributions alone. For example, a word like “painfully,” which might have a negative connotation on its own, has a large positive interaction with the word “funny” in the phrase “painfully funny.” We include more examples in the appendix.
In Figure 1, we demonstrate how interactions can help us understand one reason why a finetuned DistilBERT model outperforms a simpler model: a convolutional neural network (CNN) that gets an accuracy of 0.82 on the validation set. DistilBERT picks up on positive interactions between negation words (“not”) and negative adjectives (“bad”) that a CNN fails to fully capture.
Finally, in Figure 5, we use interaction values to reveal saturation effects: many negative adjectives describing the same noun interact positively. Although this may seem counterintuitive at first, it reflects the structure of language. If a phrase has only one negative adjective, it stands out as the word that makes the phrase negative. At some point, however, describing a noun with more and more negative adjectives makes any individual negative adjective less important towards classifying that phrase as negative.
5.2 Heart disease prediction
Here we aggregate interactions learned from many samples in a clinical dataset and use the interactions to reveal global patterns. We examine the Cleveland heart disease dataset (Detrano et al., 1989; Das et al., 2009). After preprocessing, the dataset contains 298 patients with 13 associated features, including demographic information like age and gender and clinical measurements such as systolic blood pressure and serum cholesterol. The task is to predict whether or not a patient has coronary artery disease.
We split the data into 238 patients for training (of which 109 have coronary artery disease) and 60 for testing (of which 28 have coronary artery disease). A twolayer neural network with softplus activation achieves a test accuracy of 0.87 (0.82 TPR and 0.91 TNR) and a test AUROC of 0.92. We did not extensively search over hyperparameters for this task, as our goal is not state of the art performance but rather model interpretation.
In Figure 6, we examine the interactions with a feature describing the number of major coronary arteries with calcium accumulation (0 to 3), as determined by cardiac cinefluoroscopy Detrano et al. (1986). Previous research has shown that this technique is a reliable way to gauge calcium buildup in major blood vessels, and serves as a strong predictor of coronary artery disease Detrano et al. (1986); Bartel et al. (1974); Liu et al. (2015). Our model correctly learns that more coronary arteries with evidence of calcification indicate increased risk of disease. Additionally, Integrated Hessians reveals that our model learns a negative interaction between the number of coronary arteries with calcium accumulation and female gender. This supports the wellknown phenomenon of underrecognition of heart disease in women – at the same levels of cardiac risk factors, women are less likely to have clinically manifest coronary artery disease Maas and Appelman (2010).
In addition to including more details about training procedure and dataset preprocessing, we show additional interactions and attributions in the appendix.
5.3 Drug combination response prediction
In the domain of anticancer drug combination response prediction, plotting Integrated Hessians helps us to glean biological insights into the process we are modeling. We consider one of the largest publiclyavailable datasets measuring drug combination response in acute myeloid leukemia Tyner et al. (2018). Each one of 12,362 samples consists of the measured response of a 2drug pair tested in the cancer cells of a patient. The input features are split between features describing the drug combinations (binary labels of individual drugs and their molecular targets) and features describing the cancerous cells (RNAseq expression levels for all genes). Following Hao et al. (2018), we first learn an embedding of biological pathways from individual gene expression levels in order to have a more interpretable model. Following Preuer et al. (2017), the drug features and pathway features are then input into a simple feedforward neural network (see appendix for more details).
Looking at only the firstorder attributions, we see that the presence or absence of the drug Venetoclax in the drug combination is the most important feature (see Figure 7, top left). We can also easily see that firstorder explanations are inadequate in this case – while the presence of Venetoclax is generally predictive of a more responsive drug combination, the amount of positive response to Venetoclax is predicted to vary across samples.
Integrated Hessians gives us the insight that some of this variability can be attributed to the drug Venetoclax is combined with. We can see that the model has learned a strong negative interaction between Venetoclax and Artemisinin (see Figure 7, top right). Biological interactions are known to occur between anticancer drugs, and are an area of great clinical interest to to their potential therapeutic effects. Using additional data not directly available to the model, we can determine the ground truth as to which patients actually had positive and negative biological interactions between Venetoclax and Artemisinin (see appendix for details of calculation). We see that the Integrated Hessians interaction values are significantly more negative in the group with real biological negative interactions ().
Finally, we can gain insight into the variability in the interaction values between Venetoclax and Artemisinin by plotting them against the expression level of a pathway containing cancer genes (see Figure 7, bottom). We see that patients with higher expression of this pathway tend to have a more negative interaction (subadditive response) than patients with lower expression of this pathway. Integrated Hessians helps us understand the interactions between drugs in our model, as well as what genetic factors influence this interaction.
5.4 Pulsar star prediction
In this section, we use a physics dataset to confirm that a model has learned global pattern that is visible in the training data. We utilize the HRTU2 dataset, curated by Lyon et al. (2016) and originally gathered by Keith et al. (2010). The task is to predict whether or not a particular signal measured from a radio telescope is a pulsar star or generated from radio frequency interference (e.g. background noise). The features include statistical descriptors of measurements made from the radio telescope. The dataset contains 16,259 examples generated through radio frequency interference and 1,639 examples that are pulsars. We leave a more complete description of the data and its features to the appendix, and refer the reader to Lyon (2016) for a more complete background on pulsar star detection.
We split the data into 14,318 training examples (1,365 are pulsars) and 3,580 testing examples (274 are pulsars). We train a twolayer, softplus neural network and achieve a held out test accuracy of 0.98 (0.86 TPR and 0.99 TNR). In Figure 8, we examine the interaction between two key features in the dataset: kurtosis of the integrated profile, which we abbreviate as kurtosis (IP), and standard deviation of the dispersionmeasure signaltonoise ratio curve, which we abbreviate as standard deviation (DMSNR).
The bottom of Figure 8 shows that kurtosis (IP) is a highly predictive feature, while standard deviation (DMSNR) is less predictive. However, in the range where kurtosis (IP) is roughly between 0 and 2, standard deviation (DMSNR) helps distinguish between a concentration of negative samples at standard deviation (DMSNR) 40. We can verify that the model we’ve trained correctly learns this interaction. By plotting the interaction values learned by the model against the value of kurtosis (IP), we can see a peak positive interaction for points in the indicated range and with high standard deviation (DMSNR). Interaction values show us that the model has successfully learned the expected pattern: that standard deviation (DMSNR) has the most discriminative power when kurtosis (IP) is in the indicated range.
6 Additional Related Work
Although we covered the bulk of the related work in Section 1, we mention some additional related work here. Detecting interactions  when two or more variables have a combined effect not equal to their additive effect  has a long history in statistics Southwood (1978), economics Balli and Sørensen (2013) and game theory Grabisch and Roubens (1999). In the context of machine learning, many methods have been proposed to learn interactions from data, e.g. using additive models Coull et al. (2001); Lou et al. (2013), grouplasso Lim and Hastie (2015), or trees Sorokina et al. (2008). We view these approaches as tangential to our work: they aim to learn new models to detect specific, pairwise interactions; we propose a method to explain interactions in deep networks that have already been trained. Doing so is especially important in domains where neural networks achieve state of the art performance, like natural language.
We are unaware of any work to explain interactions in neural networks other than those works mentioned in Section 1 Cui et al. (2019); Tsang et al. (2017); Greenside et al. (2018). Parallel to our work, Lundberg et al. (2020) propose an extension of feature attribution methods to feature interactions for treebased models which satisfies similar properties to the ones we propose.
7 Conclusion
In this work we propose a novel method to explain feature interactions in neural networks. The interaction values we propose have two natural interpretations: (1) as the combined effect of two features to the output of a model, and (2) as the explanation of one feature’s importance in terms of another. Our method provably satisfies commonsense axioms that previous methods do not  and unlike such previous methods, places no requirement on neural network architecture, class or data type.
Additionally, we demonstrate how to glean interactions from neural networks trained with a ReLU activation function which has no secondderivative. In accordance with recent work, we show why replacing the ReLU activation function with the softplus activation function at explanation time is both intuitive and efficient.
Finally, we perform extensive experiments to reveal the utility of our method, from understanding performance gaps between model classes to discovering patterns a model has learned on highdimensional data. We conclude that although feature attribution methods provide valuable insight into model behavior, such methods by no means end the discussion on interpretability. Rather, they encourage further work in deeper understanding model behavior.
Appendix
Appendix A Deriving Interaction Values
In this section, we expand the derivation of the Integrated Hessians formula, further discuss relevant axioms that interaction values should satisfy, and how we compute Integrated Hessians in practice.
a.1 The Integrated Hessians Formula
Here we derive the formula for Integrated Hessians from it’s definition: . We start by expanding out using the definition of Integrated Gradients:
(10) 
We consider the function , and we first assume that
(11)  
(12)  
(13)  
(14) 
where we’ve assume that the function satisfies the conditions for the Leibniz Integral Rule (so that integration and differentiation are interchangeable). These conditions require that the derivative of , and its second derivative function are continuous over in the integration region, and that the bounds of integration are constant with respect to . It’s easy to see that the bounds of integration are constant with respect to . It is also straightforward to see that common neural network activation functions  for example, , , or where is the cumulative distribution function of the normal distribution  have continuous first and second partial derivatives, which implies compositions of these functions have continuous first and second partial derivatives as well. Although this is not the case with the ReLU activation function, we discuss replacing it with softplus in the maintext.
We can proceed by plugging equation 14 into the original definition of :
(15)  
(16)  
(17) 
where all we’ve done is rearrange terms.
Deriving proceeds similarly:
(18)  
(19)  
(20) 
using the chain rule. After similar rearrangement, we can arrive at:
(21) 
The derivation for Expected Hessians simply observes that the integrals can be viewed as integrating over the product of two uniform distributions , e.g.
(22)  
(23) 
where represents the data distribution and .
a.2 Axioms Satisfied by Integrated Hessians
Self and Interaction Completeness
The proof that Integrated Hessians satisfies the axioms interaction completeness and self completeness are straightforward to show, but we include the stepbystep derivation here. First, we note that for any because . Then, by completeness of Integrated Gradients, we have that:
(24) 
Rearrangement gives us the self completeness axioms:
(25) 
Since Integrated Gradients satisfies completeness, we have:
(26) 
Making the appropriate substitution from equation 24 shows the interaction completeness axiom:
(27) 
Sensitivity
Integrated Gradients satisfies an axiom called sensitivity, which can be phrased as follows. Given an input and a baseline , if for all except where and if , then . Specifically, by completeness we know that . Intuitively, this is saying that if only one feature differs between the baseline and the input and changing that feature changes the output, then the amount the output changes should be equal to the importance of that feature.
We can extend this axiom to the interaction case by considering the case when two features differ from the baseline. We call this axiom interaction sensitivity, and can be described as follows. If an input and a baseline are equal everywhere except and , and if , then: and for all . Intuitively, this says that if the only features that differ from the baseline are and , then the difference in the output must be solely attributable to the main effects of and plus the interaction between them. This axiom holds simply by applying interaction completeness and observing that if or .
Implementation Invariance
The implementation invariance axiom, described in the original paper, states the following. For two models and such that , then for all features and all points regardless of how and are implemented. Although it seems trivial, this axiom does not necessarily hold for attribution methods that use the implementation or structure of the network in order to generate attributions. Critically, this axiom also does not hold for the interaction method proposed by Tsang et al. (2017), which looks at the first layer of a feed forward neural network. Two networks may represent exactly the same function but differ greatly in their first layer.
This axiom is trivially seen to hold for Integrated Hessians since it holds for Integrated Gradients. However, this axiom is desirable because without it, it may mean that attributions/interactions are encoding information about unimportant aspects of model structure rather than the actual decision surface of the model.
Linearity
Integrated Gradients satisfies an axiom called linearity, which can be described as follows. Given two networks and , consider the output of the weighted ensemble of the two networks . Then the attribution of the weighted ensemble equals the weighted sum of attributions for all features and samples . This axiom is desirable because it preserves linearity within a network, and allows easy computation of attributions for network ensembles.
We can generalize linearity to interactions using the interaction linearity axiom: for any and all points . Given that is composition of linear functions , in terms of the parameterized networks and , it itself is a linear function of the networks and therefore Integrated Hessians satisfies interaction linearity.
SymmetryPreserving
We say that two features and are symmetric with respect to if swapping them does not change the output of anywhere. That is, ). The original paper shows that Integrated Gradients is symmetrypreserving, that is, if and are symmetric with respect to , and if and for some input and baseline , then . We can make the appropriate generalization to interaction values: if the same conditions as above hold, then for any feature . This axiom holds since, again and are symmetrypreserving. This axiom is desirable because it says that if two features are functionally equivalent to a model, then they must interact the same way with respect to that model.
a.3 Approximating Integrated Hessians in Practice
To compute Integrated Gradients in practice, Sundararajan et al. (2017) introduce the following discrete sum approximation:
(28) 
where is the number of points used to approximate the integral. To compute Integrated Hessians, we introduce a similar discrete sum approximation:
(29) 
Typically, it is easiest to compute this quantity when and the number of samples drawn is thus a perfect square  however, when a nonsquare number of samples is desired we can generate a number of sample points from the product distribution of two uniform distributions such that the number is the largest perfect square above the desired number of samples, and index the sorted samples appropriately to get the desired number. The above formula omits the firstorder term in but it can be computed using the same principle.
Expected Hessians has a similar, if slightly easier form:
(30) 
where is the th sample from the product distribution of two uniform distributions. We find that in general that less than 300 samples are required for any given problem to approximately satisfy interaction completeness. For most problems, a number far less than 300 suffices (e.g. around 50) although this is model and data dependent: larger models and higherdimensional data generally require more samples than smaller models and lowerdimensional data.
Appendix B Effects of Smoothing ReLU Networks
b.1 Proof of Theorem 1
Theorem 2.
For a onelayer neural network with softplus\textsubscript nonlinearity, softplus\textsubscript, and input features, we can bound the number of interpolation points needed to approximate the Integrated Hessians to a given error tolerance by .
Proof.
As pointed out in Sundararajan et al. (2017) and Sturmfels et al. (2020), completeness can be used to assess the convergence of the approximation. We first show that decreasing improves convergence for Integrated Gradients. In order to accurately calculate the Integrated Gradients value for a feature , we want to be able to bound the error between the approximate computation and exact value. The exact value is given as:
(31) 
To simplify notation, we can define the partial derivative that we want to integrate over in the th coordinate as :
(32) 
Since the single layer neural network with softplus activation is monotonic along the path, the error in the approximate integral can be lower bounded by the left Riemann sum :
(33) 
and can likewise be upperbounded by the right Riemann sum :
(34) 
We can then bound the magnitude of the error between the Riemann sum and the true integral by the difference between the right and left sums:
(35) 
By the mean value theorem, we know that for some and , . Therefore:
(36) 
Rewriting in terms of the original function, we have:
(37) 
We can then consider the gradient vector of :
(38) 
where each coordinate is maximized at the zeros input vector, and takes a maximum value of . We can therefore bound the error in convergence as:
(39) 
Ignoring the dependency on path length and the magnitude of the weights of the neural network, we see that:
(40) 
This demonstrates that the number of interpolation points necessary to achieve a set error rate decreases as the activation function is smoothed (the value of is decreased). While this proof only bounds the error in the approximation of the integral for a single feature, we get the error in completeness by multiplying by an additional factor of features.
We can extend the same proof to the Integrated Hessians values. We first consider the error for estimating off diagonal terms . The true value we are trying to approximate is given as:
(41) 
For the sake of notation, we can say . Assuming that we are integrating from the allzeros baseline as suggested in Sundararajan et al. (2017), since is monotonic on either interval from the 0 baseline, we can again bound the error in the double integral by the magnitude of the difference in the left and right Riemann sums…
(42) 
(43) 
We can then again use monotonicity over the interval to say that , which gives us:
(44) 
By the mean value theorem, we know that for some , . Substituting gives us:
(45) 
We can then consider the elements of the gradient vector:
(46) 
For the univariate version of each coordinate, we can maximize the function by taking the derivative with respect to and setting it equal to 0.
(47) 
We can see that this equation holds only when = 0, and can solve it by finding the roots of this quadratic equation, which occur when . When we plug that back in, we find the absolute value of the function in that coordinate takes a maximum value of . Therefore, for a given set of fixed weights of the network, we can see that the coordinatewise maximum magnitude of , and that the number of interpolation points necessary to reach a desired level of error in approximating the double integral decreases as is decreased. Again ignoring the fixed weights and path length, the number of interpolation points necessary is bounded by:
(48) 
For the terms (main effect terms) the error will have another additive factor of . This is because there is an added term to the main effect equal to:
(49) 
When we bound the error in this approximate integral by the difference between the double left sum and double right sum, we get that:
(50) 
Following the exact same steps as in Equation 35 through Equation 39, we can then show the bound on the error of the ondiagonal terms will have an additional term that is . Due to the axiom of interaction completeness, the error bound of the entire convergence can be obtained by adding up all of the individual terms, incurring another factor of in the bound.
∎
b.2 Empirically Improves Convergence
In addition to theoretically analyzing the effects of smoothing the activation functions of a singlelayer neural network on the convergence of the approximately calculation of Integrated Gradients and Integrated Hessians, we also wanted to empirically analyze the same phenomenon in deeper networks. We assessed this by creating two networks: one with 5 hidden layers of 50 nodes, and a second with 10 hidden layers of 50 nodes. These networks were then randomly initialized using the Xavier Uniform intialization scheme (Glorot and Bengio, 2010). We created 10 samples to explain, each with 100 features drawn at random from the standard normal distribution. To evaluate the convergence of our approximate Integrated Hessians values, we plot the interaction completeness error (the difference between the sum of the Integrated Hessians value and the difference of the function output at a sample and the function output at the zeros baseline). We plot the completeness error as a fraction of the magnitude of the function output. As we decrease the value of , we smooth the activations, and we can see that the number of interpolations required to converge decreases (see Figure 9 and Figure 10). We note that the randomly initialized weights of each network are held constant and the only thing changed is the value of in the activation function.
Appendix C Details on the Sentiment Analysis Task
c.1 FineTuning DistilBERT
As mentioned in the main text, we download pretrained weights for DistilBERT, a pretrained language model introduced in Sanh et al. (2019), from the HuggingFace Transformers library Wolf et al. (2019). We finetune the model on the Stanford Sentiment Treebank dataset introduced by Socher et al. (2013). We finetune for 3 epochs using a batch size of 32 and a learning rate of 0.00003. We use a a max sequence length of 128 tokens, and the Adam algorithm for optimization Kingma and Ba (2014). We tokenize using the HuggingFace buildin tokenizer, which does so in an uncased fashion. We did not search for these hyperparameters  rather, they were the defaults presented for finetuning in the HuggingFace repository. We find that they work adequately for our purposes, and didn’t attempt to search through more hyperparameters.
c.2 Training a CNN
The convolutional neural network that we compare to in the main text is one we train from scratch on the same dataset. We randomly initialize 32dimensional embeddings and use a max sequence length of 52. First, we apply dropout to the embeddings with dropout rate 0.5. The network itself is composed of 1D convolutions with 32 filters of size 3 and 32 filters of size 8. Each filter size is applied separately to the embedding layer, after which max pooling with a stride of 2 is applied and then the output of both convolutions is concatenated together and fed through a dropout layer with a dropout rate of 0.5 during training. A hidden layer of size 50 follows the dropout, finally followed by a linear layer generating a scalar prediction that the sigmoid function is applied to.
We train with a batch size of 128 for 2 epochs and use a learning rate of 0.001. We optimize using the Adam algorithm with the default hyperparameters Kingma and Ba (2014). Since this model was not pretrained on a large language corpus and lacks the expressive power of a deep transformer, it is unable to capture patterns like negation that a finetuned DistilBERT does.
c.3 Generating Attributions and Interactions
In order to generate attributions and interactions, we use Integrated Gradients and Integrated Hessians with the zeroembedding  the embedding produced by the all zeros vector, which normally encodes the padding token  as a baseline. Because embedding layers are not differentiable, we generate attributions and interactions to the word embeddings and then sum over the embedding dimension to get wordlevel attributions and interactions, as done in Sundararajan et al. (2017). When computing attributions and interactions, we use 256 background samples. Because DistilBERT uses the GeLU activation function Ramachandran et al. (2017), which has continuous first and second partial derivatives, there is no need to use the softplus replacement. When we plot interactions, we avoid plotting the maineffect terms in order to better visualize the interactions between words.
c.4 Additional Examples of Interactions
Here we include additional examples of interactions learned on the sentiment analysis task. First we expand upon the idea of saturation in natural language, displayed in Figure 11. We display interactions learned by a finetuned DistilBERT on the following sentences: “a bad movie” (negative with 0.9981 confidence), “a bad, terrible movie” (negative with 0.9983 confidence), “a bad, terrible, awful movie” (negative with 0.9984 confidence) and “a bad, terrible, awful, horrible movie” (negative with 0.9984 confidence). The confidence of the network saturates: a network output only gets so negative before it begins to flatten. However, the number of negative adjectives in the sentence increases. This means a sensible network would spread the same amount of credit (because the attributions sum to the saturated output) across a larger number of negative words, which is exactly what DistilBERT does. However, this means that each word gets less negative attribution than it would if it was on its own. Thus, the negative words have positive interaction effects, which is exactly what we see from the figure.
In Figure 12, we give another example of the full interaction matrix on a sentence from the validation set. In Figure 13, we give an example of how explaining the importance of a particular word can understand whether that word is important because of its main effect or because of its surrounding context. We show additional examples from the validation set in Figures 14, 15, 16, 17, 18. We note that while some interactions make intuitive sense to humans (“better suited” being negative or “good script” being positive), there are many other examples of interactions that are less intuitive. These interactions may indicate that the Stanford Sentiment Treebank dataset does not fully capture the expressive power of language (e.g. it doesn’t have enough samples to fully represent all of the possible interactions in language), or it may indicate that the model has learned higher order effects that cannot be explained by pairwise interactions alone.
Appendix D Details on the Heart Disease Prediction Task
d.1 Data Description
As mentioned in the main text, the dataset we use has 13 associated features. The list of features, which we reproduce here, is from Detrano et al. (1989), the original paper introducing the dataset:

Age of patient (mean: 54.5 years standard deviation: 9.0)

Gender (202 male, 96 female)

Resting systolic blood pressure (131.6 mm Hg 17.7)

Cholesterol (246.9 mg/dl 51.9)

Whether or not a patient’s fasting blood sugar was above 120 mg/dl (44 yes)

Maximum heart rate achieved exercise (149.5 bpm 23.0)

Whether or not a patient has exerciseinduced angina (98 yes)

Excerciseinduced STsegment depression (1.05 mm 1.16)

Number of major vessels appearing to contain calcium as revealed by cinefluoroscopy (175 patients with 0, 65 with 1, 38 with 2, 20 with 3)

Type of pain a patient experienced if any (49 experienced typical anginal pain, 84 experienced atypical anginal pain, 23 experienced nonanginal pain and 142 patients experienced no chest pain)

Slope of peak exercise ST segment (21 patients had upsloping segments, 138 had flat segments, 139 had downsloping segments)

Whether or not a patient had thallium defects as revealed by scintigraphy (2 patients with no information available, 18 with fixed defects, 115 with reversible defects and 163 with no defects)

Classification of resting electrocardiogram (146 with normal resting ecg, 148 with an STT wave abnormality, and 4 with probable or definite left centricular hypertrophy)
d.2 Training Details
On this task, we use a two layer neural network with 128 and 64 hidden units, respectively, with softplus activation after each layer. We optimize using gradient descent (processing the entire training set in a single batch) with an initial learning rate of 0.1 that decays exponentially with a rate 0.99 after each epoch. We use nesterov momentum with (Sutskever et al., 2013). After training for 200 epochs, the network achieves a heldout accuracy of 0.8667 with 0.8214 true positive rate and 0.9062 true negative rate. We note that the hyperparameters chosen here were not carefully tuned on a validation set  they were simply those that seemed converge to a reasonable performance on the training set. Our focus is not state of the art prediction or comparing model performances, but rather interpreting patterns a reasonable model learns.
d.3 Generating Attributions and Interactions
To generate attributions and interactions for this dataset, we use Expected Gradients and Expected Hessians with the training set forming the background distribution. We use 200 samples to compute both attributions and interactions, although we note this number is probably larger than necessary but was easy to compute due to the small size of the dataset.
d.4 Additional Examples of Attributions and Interactions
Figure 19 shows which features were most important towards predicting heart disease aggregated over the entire dataset, as well as the trend of importance values. Interestingly, the model learns some strangely unintuitive trends: if a patient doesn’t experience chest pain, they are more likely to have heart disease than if they experience anginal chest pain! This could indicate problems with the way certain features were encoded, or perhaps dataset bias. Figure 20 demonstrates an interaction learned by the network between maximum heart rate achieved and gender, and Figure 21 demonstrates an interaction between exerciseinduced STsegment depression and number of major vessels appearing to contain calcium.
Appendix E Details for anticancer drug combination response prediction
e.1 Data description
As mentioned in the main text, our dataset consists of 12,362 samples (available from http://www.vizome.org/). Each sample consists of the measured response of a 2drug pair tested in the cancer cells of a patient Tyner et al. (2018). The 2drug combination was described with both a drug identity indicator and a drug target indicator. For each sample, the drug identity indicator is a vector where each element represents one of the 46 anticancer drugs present in the data, and takes a value of if the corresponding drug is not present in the combination and a value of if the corresponding drug is present in the combination. Therefore, for each sample, will have 44 elements equal to 0 and 2 elements equal to 1. This is the most compact possible representation for the 2drug combinations. The drug target indicator is a vector where each element represents one of the 112 unique molecular targets of the anticancer drugs in the dataset. Each entry in this vector is equal to if neither drug targets the given molecule, equal to if one of the drugs in the combination targets the given molecule, and equal to if both drugs target the molecule. The targets were compiled using the information available on DrugBank (Wishart et al., 2018). The ex vivo samples of each patient’s cancer was described using gene expression levels for each gene in the transcriptome, as measured by RNAseq, . Before training, the data was split into two parts – 80% of samples were used for model training, and an additional 20% were used as a heldout validation set to determine when the model had been trained for a sufficient number of epochs.
e.2 RNAseq preprocessing
The cancerous cells in each sample were described using RNAseq data – measurements of the expression level of each gene in the sample. We describe here the preprocessing steps used to remove batch effects while preserving biological signal. We first converted raw transcript counts to fragments per kilobase of exon model per million mapped reads (FPKM), a measure that is known to better reflect the molar amount of each transcript in the original sample than raw counts. FPKM accounts for this by normalizing the counts for different genes according to the length of transcript, as well as for the total number of reads included in the sample (Mortazavi et al., 2008). The equation for FPKM is given as:
(51) 
where is the vector containing the number of raw counts for a particular transcript across all samples, is the effective length of that transcript, and represents the total number of counts. After converting raw counts to FPKM, we opt to consider only the proteincoding part of the transcriptome by removing all nonproteincoding transcripts from the dataset. Proteincoding transcripts were determined according to the list provided by the HUGO Gene Nomenclature Committee (https://www.genenames.org/download/statisticsandfiles/). In addition to nonproteincoding transcripts, we also removed any transcript that was not observed in of samples. Transcripts are then transformed and made 0mean unit variance. Finally, the ComBat tool (a robust empirical Bayes regression implemented as part of the sva R package) was used to correct for batch effects (Leek and Storey, 2007).
e.3 Model and training description
To model the data, we combined the successful approaches of Preuer et al. (2017) and Hao et al. (2018). Our network architecture is a simple feedforward network (Figure 22), as in Preuer et al. (2017), where there were two hidden layers of 500 and 250 nodes respectively, both with Tanh activation. In order to improve performance and interpretability, we followed Hao et al. (2018) in learning a pathwaylevel embedding of the gene expression data. The RNAseq data, , was sparsely connected to a layer of nodes, where each node corresponded to a single pathway from KEGG, BioCarta, or Reactome (Kanehisa, 2002; Nishimura, 2001; Croft et al., 2014). We made this embedding nonlinear by following the sparse connections with a Tanh activation function. The nonlinear pathway embeddings were then concatenated to the drug identity indicators and the drug target indicators, and these served as inputs to the densely connected layers.We trained the network to optimize a mean squared error loss function, and used the Adam optimizer in PyTorch with default hyperparameters and a learning rate equal to (Kingma and Ba, 2014). We stopped the training when mean squared error on the heldout validation set failed to improve over 10 epochs, and found that the network reached an optimum at epochs. For the sake of easier calculation and more humanintuitive attribution, we attribute the model’s output to the layer with the pathway embedding and drug inputs, rather than to the raw RNAseq features and drug inputs (see Figure 22).
For this experiment, we calculated all explanations and interactions using the Integrated Gradients and Integrated Hessians approach using the all zeros vector as reference and interpolation points.
e.4 Biological interaction calculation
To evaluate how well the interactions detected by Integrated Hessians match with the ground truth for biological drugdrug interactions in this dataset, we can use additional single drug response data that our model was not given access to in order to calculate biological synergy. Drug synergy is the degree of extraadditive or subadditive response observed when two drugs are combined as compared to the additive response that would be expected if there were no interaction between the two compounds. The drug response for a single drug is measured as , or the dose of that single drug necessary to kill half of the cells in an ex vivo sample. The drug response for a drug combination is measured as , or the dose of an equimolar combination of two drugs necessary to kill half of the cells in an ex vivo sample. The drug synergy between two drugs and can be calculated using the , or combination index:
(52) 
For , a value greater than 1 indicates antisynergy (negative interaction) while a value less than 1 indicates synergy (positive interaction). While our model was trained solely to predict , we can see how well the model learned true biological interactions by using the additional single drug response data to calculate synergy for particular samples. As described in the maintext, when we calculated values for all samples in which the combination of Venetoclax and Artemisinin was tested, then binarized the samples into synergistic and antisynergistic, we can see that the Integrated Hessians values were higher in the truly synergistic group than in the truly antisynergistic group (). We note that this is particularly remarkable, given that the model had no access to single drug data whatsoever.
Appendix F Details on the pulsar star prediction task
f.1 Data description
The dataset we use has 4 statistical descriptors  mean, standard deviation, skewness and kurtosis  of two measurements relating to pulsar stars: the integrated pulse profile (IP) and the dispersionmeasure signaltonoise ratio curve (DMSNR), for a total of 8 features. The integrated pulse profile measures how much signal the supposed pulsar star gives off as a function of phase of the pulsar: as pulsars rotate, they emit radiation from their magnetic poles, which sweep over the earth periodically. We can measure the radiation over time using a radio telescope and aggregating measurements over phase to get the integrated pulse profile. Signals that are pulsars stars should in theory have stronger, more peaked integrated pulse profiles than those generated from radio frequency interference. The DMSNR curve measures how much phase correction changes the signaltonoise ratio in the measured signal. Since pulsars are far away, their radio emissions get dispersed as they travel from the star to earth: low frequencies get dispersed more than high frequencies (e.g. they arrive later). Phase correction attempts to resync the frequencies; however, no amount of phase correction should help peak a signal if the signal was generated from radio frequency interference rather than a legitimate pulsar.
f.2 Training Details
On this task, we use a twolayer neural network with 32 hidden units in both layers and the softplus activation function after each layer. We optimize using stochastic gradient descent with a batch size of 256. We use an initial learning rate of 0.1 that decays with a rate of 0.96 every batch. We use nesterov momentum as well with Sutskever et al. (2013). We train for 10 epochs and use a classweight ratio of 1:3 negative to positive to combat the imbalance in the training dataset. Again, we note that these hyperparameters are not necessarily optimal but were simply chosen because they produced reasonable convergence on the training set.
f.3 Generating Attributions and Interactions
To generate attributions and interactions for this dataset, we use Expected Gradients and Expected Hessians with the training set forming the background distribution. We use 200 samples to compute both attributions and interactions, although, as also noted on the previous section about the heart disease task, 200 samples was probably larger than necessary.
Footnotes
 Code available at https://github.com/suinleelab/path_explain
 In the case of multioutput models, such as multiclass classification problems, we assume the function is indexed into the correct output class.
 We discuss the special case of the ReLU activation function in Section 3.
 This performance does not represent state of the art, nor is sentiment analysis representative of the full complexity of existing language tasks. However, our focus in this paper is on explanation and this task is easy to finetune without needing to extensively search over hyperparameters.
References
 Towards better understanding of gradientbased attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104. Cited by: §2.3.
 Interaction effects in econometrics. Empirical Economics 45 (1), pp. 583–603. Cited by: §6.
 The significance of coronary calcification detected by fluoroscopy: a report of 360 patients. Circulation 49 (6), pp. 1247–1253. Cited by: §5.2.
 Layerwise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63–71. Cited by: §1, §2.3.
 On identifiability in transformers. arXiv preprint arXiv:1908.04211. Cited by: §5.1.
 Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. arXiv preprint arXiv:2001.08298. Cited by: §1.
 Simple incorporation of interactions into additive models. Biometrics 57 (2), pp. 539–545. Cited by: §6.
 The reactome pathway knowledgebase. Nucleic acids research 42 (D1), pp. D472–D477. Cited by: §E.3.
 Recovering pairwise interactions using neural networks. arXiv preprint arXiv:1901.08361. Cited by: §1, §1, §4, §6.
 Effective diagnosis of heart disease through neural networks ensembles. Expert systems with applications 36 (4), pp. 7675–7680. Cited by: §5.2.
 International application of a new probability algorithm for the diagnosis of coronary artery disease. The American journal of cardiology 64 (5), pp. 304–310. Cited by: §D.1, §5.2.
 Cardiac cinefluoroscopy as an inexpensive aid in the diagnosis of coronary artery disease. The American Journal of Cardiology 57 (13), pp. 1041 – 1046. External Links: ISSN 00029149, Document, Link Cited by: §5.2.
 Bert: pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: §1, §5.1.
 Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, pp. 13567–13578. Cited by: §3, §3.
 Visualizing higherlayer features of a deep network. University of Montreal 1341 (3), pp. 1. Cited by: §1.
 Learning explainable models using attribution priors. arXiv preprint arXiv:1906.10670. Cited by: §2.3.
 Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429–3437. Cited by: §2.3.
 Net2vec: quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730–8738. Cited by: §1.
 Ethics of artificial intelligence in radiology: summary of the joint European and North American multisociety statement. Radiology 293 (2), pp. 436–440. Cited by: §1.
 Interpreting recurrent and attentionbased neural models: a case study on natural language inference. arXiv preprint arXiv:1808.03894. Cited by: §5.1.
 Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256. Cited by: §B.2.
 An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory 28 (4), pp. 547–565. Cited by: §6.
 Discovering epistatic feature interactions from neural network models of regulatory dna sequences. Bioinformatics 34 (17), pp. i629–i637. Cited by: §1, §6.
 PASNet: pathwayassociated sparse deep neural network for prognosis prediction from highthroughput data. BMC bioinformatics 19 (1), pp. 1–13. Cited by: §E.3, §5.3.
 Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §1.
 Attention is not explanation. arXiv preprint arXiv:1902.10186. Cited by: §5.1.
 The kegg database. In Novartis Foundation Symposium, pp. 91–100. Cited by: §E.3.
 Segment integrated gradients: better attributions through regions. arXiv preprint arXiv:1906.02825. Cited by: §2.3.
 The high time resolution universe pulsar survey–i. system configuration and initial discoveries. Monthly Notices of the Royal Astronomical Society 409 (2), pp. 619–627. Cited by: §5.4.
 Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). arXiv preprint arXiv:1711.11279. Cited by: §1.
 Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. Cited by: §5.1.
 The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267–280. Cited by: §2.3.
 Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §C.1, §C.2, §E.3.
 Many faces of feature importance: comparing builtin and posthoc feature importance in text classification. arXiv preprint arXiv:1910.08534. Cited by: §5.1.
 Interactive visualization and manipulation of attentionbased neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 121–126. Cited by: §5.1.
 Capturing heterogeneity in gene expression studies by surrogate variable analysis. PLOS Genetics 3 (9), pp. 1–12. External Links: Link, Document Cited by: §E.2.
 Learning interactions via hierarchical grouplasso regularization. Journal of Computational and Graphical Statistics 24 (3), pp. 627–654. Cited by: §6.
 A structured selfattentive sentence embedding. arXiv preprint arXiv:1703.03130. Cited by: §5.1.
 Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286. Cited by: §5.1.
 Current understanding of coronary artery calcification. Journal of geriatric cardiology: JGC 12 (6), pp. 668. Cited by: §5.2.
 Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 623–631. Cited by: §6.
 A unified approach to interpreting model predictions. In Advances in neural information processing systems, pp. 4765–4774. Cited by: §1, §1, §2.3.
 From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence 2 (1), pp. 56–67. External Links: Document, ISBN 25225839, Link Cited by: §6.
 Fifty years of pulsar candidate selection: from simple filters to a new principled realtime classification approach. Monthly Notices of the Royal Astronomical Society 459 (1), pp. 1104–1123. Cited by: §5.4.
 Why are pulsars hard to find?. Ph.D. Thesis, The University of Manchester (United Kingdom). Cited by: §5.4.
 Gender differences in coronary heart disease. Netherlands Heart Journal 18 (12), pp. 598–603. Cited by: §5.2.
 Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188–5196. Cited by: §1.
 Mapping and quantifying mammalian transcriptomes by RNASeq. Nature Methods 5, pp. 621. External Links: Link Cited by: §E.2.
 BioCarta. Biotech Software & Internet Report: The Computer Software Journal for Scient 2 (3), pp. 117–120. Cited by: §E.3.
 Feature visualization. Distill 2 (11), pp. e7. Cited by: §1.
 The building blocks of interpretability. Distill 3 (3), pp. e10. Cited by: §1.
 Deep contextualized word representations. arXiv preprint arXiv:1802.05365. Cited by: §5.1.
 DeepSynergy: predicting anticancer drug synergy with Deep Learning. Bioinformatics 34 (9), pp. 1538–1546. External Links: ISSN 13674803, Document, Link, https://academic.oup.com/bioinformatics/articlepdf/34/9/1538/25120818/btx806.pdf Cited by: §E.3, §5.3.
 Magix: model agnostic globally interpretable explanations. arXiv preprint arXiv:1706.07160. Cited by: §1.
 Searching for activation functions. arXiv preprint arXiv:1710.05941. Cited by: §C.3.
 ” Why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144. Cited by: §1.
 DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Cited by: §C.1, §5.1.
 Meaningful information and the right to explanation. International Data Privacy Law 7 (4), pp. 233–242. External Links: ISSN 20443994, Document, Link, https://academic.oup.com/idpl/articlepdf/7/4/233/22923065/ipx022.pdf Cited by: §1.
 Is attention interpretable?. arXiv preprint arXiv:1906.03731. Cited by: §5.1.
 Regularization learning networks: deep learning for tabular datasets. In Advances in Neural Information Processing Systems, pp. 1379–1389. Cited by: §1.
 Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine LearningVolume 70, pp. 3145–3153. Cited by: §1, §2.3.
 Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642. Cited by: §C.1, §5.1.
 Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference on Machine learning, pp. 1000–1007. Cited by: §6.
 Substantive theory and statistical interaction: five models. American Journal of Sociology 83 (5), pp. 1154–1203. Cited by: §6.
 Visualizing the impact of feature attribution baselines. Distill 5 (1), pp. e22. Cited by: §B.1, §2.3.
 Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine LearningVolume 70, pp. 3319–3328. Cited by: §A.3, §B.1, §B.1, §C.3, Explaining Explanations, §1, §1, §2.1, §2.2, §2.3, §2.
 A note about: local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv:1806.04205. Cited by: §2.3.
 LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association, Cited by: §5.1.
 On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139–1147. Cited by: §D.2, §F.2.
 Learning global additive explanations for neural nets using model distillation. arXiv preprint arXiv:1801.08640. Cited by: §1.
 Sanity checks for saliency metrics. External Links: 1912.01451 Cited by: §1.
 Detecting statistical interactions from neural network weights. arXiv preprint arXiv:1705.04977. Cited by: §A.2.3, §1, §6.
 Functional genomic landscape of acute myeloid leukaemia. Nature 562 (7728), pp. 526–531. Cited by: §E.1, §5.3.
 Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008. Cited by: §5.1.
 Attentionbased lstm for aspectlevel sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 606–615. Cited by: §5.1.
 DrugBank 5.0: a major update to the drugbank database for 2018. Nucleic acids research 46 (D1), pp. D1074–D1082. Cited by: §E.1.
 HuggingFace’s transformers: stateoftheart natural language processing. ArXiv abs/1910.03771. Cited by: §C.1, §5.1.
 Beyond sparsity: tree regularization of deep models for interpretability. In ThirtySecond AAAI Conference on Artificial Intelligence, Cited by: §1.