A Symbolic Approach to Explaining Bayesian Network Classifiers
Abstract
We propose an approach for explaining Bayesian network classifiers, which is based on compiling such classifiers into decision functions that have a tractable and symbolic form. We introduce two types of explanations for why a classifier may have classified an instance positively or negatively and suggest algorithms for computing these explanations. The first type of explanation identifies a minimal set of the currently active features that is responsible for the current classification, while the second type of explanation identifies a minimal set of features whose current state (active or not) is sufficient for the classification. We consider in particular the compilation of Naive and LatentTree Bayesian network classifiers into Ordered Decision Diagrams (ODDs), providing a context for evaluating our proposal using case studies and experiments based on classifiers from the literature.
A Symbolic Approach to Explaining Bayesian Network Classifiers
Andy Shih and Arthur Choi and Adnan Darwiche Computer Science Department University of California, Los Angeles {andyshih,aychoi,darwiche}@cs.ucla.edu
1 Introduction
Recent progress in artificial intelligence and the increased deployment of AI systems have led to highlighting the need for explaining the decisions made by such systems, particularly classifiers; see, e.g., [?; ?; ?; ?].^{1}^{1}1It is now recognized that opacity, or lack of explainability is “one of the biggest obstacles to widespread adoption of artificial intelligence” (The Wall Street Journal, August 10, 2017). For example, one may want to explain why a classifier decided to turn down a loan application, or rejected an applicant for an academic program, or recommended surgery for a patient. Answering such why? questions is particularly central to assigning blame and responsibility, which lies at the heart of legal systems and may be required in certain contexts.^{2}^{2}2See, for example, the EU general data protection regulation, which has a provision relating to explainability, https://www.privacyregulation.eu/en/r71.htm.
In this paper, we propose a symbolic approach to explaining Bayesian network classifiers, which is based on the following observation. Consider a classifier that labels a given instance either positively or negatively based on a number of discrete features. Regardless of how this classifier is implemented, e.g., using a Bayesian network, it does specify a symbolic function that maps features into a yes/no decision (yes for a positive instance). We refer to this function as the classifier’s decision function since it unambiguously describes the classifier’s behavior, independently of how the classifier is implemented. Our goal is then to obtain a symbolic and tractable representation of this decision function, to enable symbolic and efficient reasoning about its behavior, including the generation of explanations for its decisions. In fact, [?] showed how to compile the decision functions of naive Bayes classifiers into a specific symbolic and tractable representation, known as Ordered Decision Diagrams (ODDs). This representation extends Ordered Binary Decision Diagrams (OBDDs) to use multivalued variables (discrete features), while maintaining the tractability and properties of OBDD [?; ?; ?].
We show in this paper how compiling decision functions into ODDs can facilitate the efficient explanation of classifiers and propose two types of explanations for this purpose.
The first class of explanations we consider are minimumcardinality explanations. To motivate these explanations, consider a classifier that has diagnosed a patient with some disease based on some observed test results, some of which were positive and others negative. Some of the positive test results may not be necessary for the classifier’s decision: the decision would remain intact if these test results were negative. A minimumcardinality explanation then tells us which of the positive test results are the culprits for the classifier’s decision, i.e., a minimal subset of the positive test results that is sufficient for the current decision.
The second class of explanations we consider are primeimplicant explanations. These explanations answer the following question: what is the smallest subset of features that renders the remaining features irrelevant to the current decision? In other words, which subset of features—when fixed—would allow us to arbitrarily toggle the values of other features, while maintaining the classifier’s decision?
This paper is structured as follows. In Section 2, we review the compilation of naive Bayes classifiers into ODDs, and propose a new algorithm for compiling latenttree classifiers into ODDs. In Section 3, we introduce minimumcardinality explanations, propose an algorithm for computing them, and provide a case study on a realworld classifier. In Section 4, we do the same for primeimplicant explanations. In Section 5, we discuss the relationship between the two types of explanations and show that they coincide for monotone classifiers. We then follow by a discussion of related work in Section 6 and finally close in Section 7.
2 Compiling Bayesian Network Classifiers
Consider Figure 1 which depicts a naive Bayes classifier for detecting pregnancy. Given results for the three tests, if the probability of pregnancy passes a given threshold (say ), we would then obtain a “yes” decision on pregnancy.
Figure 2 depicts the decision function of this classifier, in the form of an Ordered Binary Decision Diagram (OBDD). Given some test results, we make a corresponding decision on pregnancy by simply navigating the OBDD. We start at the root, which is labeled with the Urine (U) test. Depending on the outcome of this test, we follow the edge labeled positive, or the edge labeled negative. We repeat for the test labeled at the next node. Eventually, we reach a leaf node labeled “yes” or “no,” which provides the resulting classification.
The decisions rendered by this OBDD are guaranteed to match those obtained from the naive Bayes classifier. We have thus converted a probabilistic classifier into an equivalent classifier that is symbolic and tractable. We will later see how this facilitates the efficient generation of explanations.
We will later discuss compiling Bayesian network classifiers into ODDs, after formally treating classifiers and ODDs.
2.1 Bayesian Network Classifiers
A Bayesian network classifier is a Bayesian network containing a special set of variables: a single class variable and feature variables . The class is usually a root in the network and the features are usually leaves. In this paper, we assume that the class variable is binary, with two values and that correspond to positive and negative classes, respectively (i.e., “yes” and “no” decisions). An instantiation of variables is denoted and called an instance. A Bayesian network classifier specifying probability distribution will classify an instance positively iff , where is called the classification threshold.
Definition 1 (Decision Function)
Suppose that we have a Bayesian network classifier with features , class variable and a threshold . Let be a function that maps instances into . We say that is the classifier’s decision function iff
Instance is positive if and negative if .
The naive Bayes classifier is a special type of a Bayesian network classifier, where edges extend from the class to features (no other nodes or edges). Figure 1 depicted a naive Bayes classifier. A latenttree classifier is a treestructured Bayesian network, whose root is the class variable and whose leaves are the features.
2.2 Monotone Classifiers
The class of monotone classifiers is relevant to our discussion, particularly when relating the two types of explanations we shall propose. We will define these classifiers next, while assuming binary features to simplify the treatment. Intuitively, a monotone classifier satisfies the following. A positive instance remains positive if we flip some of its features from to . Moreover, a negative instance remains negative if we flip some of its features from to .
More formally, consider two instances and . We write to mean: the features set to in is a subset of those set to in . Monotone classifiers are then characterized by the following property of their decision functions, which is wellknown in the literature on Boolean functions.
Definition 2
A decision function is monotone iff
One way to read the above formal definition is as follows. If the positive features in instance contain those in instance , then instance must be positive if instance is positive.
It is generally difficult to decide whether a Bayesian network classifier is monotone; see, e.g., [?]. However, if the decision function of the classifier is an OBDD, then monotonicity can be decided in time quadratic in the OBDD size [?].
2.3 Ordered Decision Diagrams
An Ordered Binary Decision Diagram (OBDD) is based on an ordered set of binary variables . It is a rooted, directed acyclic graph, with two sinks called the sink and sink. Every node (except the sinks) in the OBDD is labeled with a variable with two outgoing edges, one labeled and the other labeled . If there is an edge from a node labeled to a node labeled , then . An OBDD is defined over binary variables, but can be extended to discrete variables with arbitrary values. This is called an ODD: a node labeled with variable has one outgoing edge for each value of variable . Hence, an OBDD/ODD can be viewed as representing a function that maps instances into . Figure 2 depicted an OBDD. Note: in this paper, we use positive/yes/ and negative/no/ interchangeably.
An OBDD is a tractable representation of a function as it can be used to efficiently answer many queries about the function. For example, one can in linear time count the number of positive instances (i.e., ), called the models of . One can also conjoin, disjoin and complement OBDDs efficiently. This tractability, which carries over to ODDs, will be critical for efficiently generating explanations. For more on OBDDs, see [?; ?].
2.4 Compiling Decision Functions
[?] proposed an algorithm for compiling a naive Bayes classifier into an ODD, while guaranteeing an upper bound on the time of compilation and the size of the resulting ODD. In particular, for a classifier with features, the compiled ODD has a number of nodes that is bounded by and can be obtained in time . Here, is the maximum number of values that a variable may have. The actual time and space complexity can be much less, depending on the classifier’s parameters and variable order used for the ODD (as observed experimentally).
The algorithm is based on the following insights. Let be all features. Observing features leads to another naive Bayes classifier, with features and an adjusted class prior. Consider now a decision tree over features and a node in the tree that was reached by a partial instantiation . We annotate this node with the corresponding naive Bayes classifier found by observing , and then merge nodes with equivalent classifiers—those having equivalent decision functions—as described by [?]. Implementing this idea carefully leads to an ordered decision diagram (ODD) with the corresponding bounds.^{3}^{3}3[?] uses a sophisticated, but conceptually simple, technique for identifying equivalent classifiers.
Algorithm 1 is a simpler variation on the algorithm of [?]; it has the same complexity bounds, but may be less efficient in practice. It uses procedure , which expands the partial decision graph by a feature , then merges nodes that correspond to equivalent classifiers.
Using this procedure, we propose Algorithm 2 for compiling a latenttree classifier into an ODD. Here’s the key insight. Let be a node in a latenttree classifier where all features outside have been observed, and let be a child of . Observing all features under leads to a new latenttree classifier without the subtree rooted at and an adjusted class prior. Algorithm 2 uses this observation by iteratively choosing a node and then shrinking the classifier size by instantiating the features under , allowing us to compile an ODD in a fashion similar to [?]. The specific choice of internal nodes by Algorithm 2 leads to the following complexity.
Theorem 1
Given a latenttree classifier with variables, each with at most values, the ODD computed by Algorithm 2 has size and can be obtained in time .
If one makes further assumptions about the structure of the latent tree (e.g., if the root has children, and each child of the root has features), then one obtains the size bound of and time bound of for naive Bayes classifiers. We do not expect a significantly better upper bound on the time complexity due to the following result.
Theorem 2
Given a naive Bayes classifier , compiling an ODD representing its decision function is NPhard.
3 Minimum Cardinality Explanations
We now consider the first type of explanations for why a classifier makes a certain decision. These are called minimumcardinality explanations or MCexplanations. We will first assume that the features are binary and then generalize later.
Consider two instances and . As we did earlier, we write to mean: the features set to in are a subset of those set to in . We define analogously. Moreover, we write to mean: the count of features in is no greater than their count in . We define analogously.
Definition 3 (MCExplanation)
Let be a given decision function. An MCexplanation of a positive instance is another positive instance such that and there is no other positive instance where . An MCexplanation of a negative instance is another negative instance such that and there is no other negative instance where .
Intuitively, an MCexplanation of a positive decision answers the question: which positive features of instance are responsible for this decision? Similarly for the MCexplanation of a negative decision : which negative features of instance are responsible for this decision? MCexplanations are not necessarily unique as we shall see later. However, MCexplanations of positive decisions must all have the same number of features, and those for negative decisions must all have the same number of features.
MCexplanations are perhaps best illustrated using a monotone classifier. As a running example, consider a (monotone) classifier for deciding whether a student will be admitted to a university. The class variable is () and the features of an applicant are:

(): has prior work experience.

(): did not apply before.

(): passed the entrance exam.

(): has met the university’s expected GPA.
All variables are either positive (+) or negative ().
Consider a naive Bayes classifier with the following false positive and false negative rates:
feature  

To completely specify the naive Bayes classifier, we also need the prior probability of admission, which we assume to be . Moreover, we use a decision threshold of , admitting an applicant if . Note that with the above false positive and false negative rates, a positively observed feature will increase the probability of a positive classification, while a negatively observed feature will increase the probability of a negative classification (hence, the classifier is monotone).
MCexplanations  

        0.0002   
(  + +) ( +  +) ( + + ) (+  + ) (+ +  ) 
      +  0.0426    (  + +) ( +  +) 
    +    0.0006    (  + +) ( + + ) (+  + ) 
    +  +  0.1438    (  + +) 
  +      0.0016    ( +  +) ( + + ) (+ +  ) 
  +    +  0.2933    ( +  +) 
  +  +    0.0060    ( + + ) 
  +  +  +  0.6105  +  ( + + +) 
+        0.0354    (+ +  ) (+  + ) 
+      +  0.9057  +  (+   +) 
+    +    0.1218    (+  + ) 
+    +  +  0.9732  +  (+   +) 
+  +      0.2552    (+ +  ) 
+  +    +  0.9890  +  (+   +) 
+  +  +    0.5642  +  (+ + + ) 
+  +  +  +  0.9971  +  (+   +) 
Table 1 depicts the decision function for this naive Bayes classifier, with MCexplanations for all instances.
Consider, for example, a student (+ + + +) who was admitted by this decision function. There is a single MCexplanation for this decision, (+   +), with cardinality . According to this explanation, work experience and a good GPA were the reasons for admission. That is, the student would still have been admitted even if they have applied before and did not pass the entrance exam.
For another example, consider a student (   +) who was rejected. There are two MCexplanations for this decision. The first, (  + +), says that the student would not have been admitted, even if they passed the entrance exam. The second explanation, ( +  +), says that the student would not have been admitted, even if they were a firsttime applicant.
Finally, we remark that while MCexplanations are more intuitive for monotone classifiers, they also apply to classifiers that are not monotone, as we shall see in Section 3.2.
3.1 Computing MCExplanations
We will now present an efficient algorithm for computing the MCexplanations of a decision, assuming that the decision function has a specific form. Our treatment assumes that the decision function is represented as an OBDD, but it actually applies to a broader class of representations which includes OBDDs as a special case. More on this later.
Our algorithm uses a key operation on decision functions.
Definition 4 (Cardinality Minimization)
For , the minimization of decision function is another decision function defined as follows: iff (a) and (b) for every .
The minimization of decision function renders positive decisions only on the positive instances of having a minimal number of features. Similarly, the minimization of decision function renders positive decisions only on the positive instances of having a minimal number of features. Cardinality minimization was discussed and employed for other purposes in [?; ?].
Algorithm 3 computes the MCexplanations of a decision . The set of computed explanations is encoded by another decision function . In particular, iff is an MCexplanation of decision .
Suppose we want to compute the MCexplanations of a positive decision . The algorithm will first find the portion of instance with variables set to . It will then conjoin^{4}^{4}4Conjoining with leads to a function such that iff and is compatible with . with and minimize the result. The obtained decision function encodes the MCexplanations in this case.
An OBDD can be complemented and conjoined with a variable instantiation in linear time. It can also be minimized in linear time. This leads to the following complexity for generating MCexplanations based on OBDDs.
Theorem 3
When the decision function is represented as an OBDD, the time and space complexity of Algorithm 3 is linear in the size of , while guaranteeing that the output function is also an OBDD.
Given OBDD properties, one can count MCexplanations in linear time, and enumerate each in linear time.^{5}^{5}5Minimization, conjoin, and model enumeration are all linear time operations on DNNFs, which is a superset of OBDDs [?; ?]. Moreover, where we read as “isasubclassof”. Hence, DNNFs, dDNNFs and SDDs could have been used for supporting MCexplanations, except that we would need a different algorithm for compiling classifiers. Moreover, beyond OBDDs, only SDDs support complementation in linear time. Hence, efficiently computing MCexplanations of negative decision requires that we efficiently complement the decision functions represented by DNNFs or dDNNFs.
3.2 Case Study: Votes Classifier
We now consider the Congressional Voting Records () from the UCI machine learning repository [?]. This dataset consists of 16 key votes by Congressmen of the U.S. House of Representatives. The class label is the party of the Congressman (positive if Republican and negative if Democrat). A naive Bayes classifier trained on this dataset obtains 91.0% accuracy. We compiled this classifier into an OBDD, which has a size of 630 nodes.
The following Congressman from the dataset voted on all 16 issues and was classified correctly as a Republican:
(0 1 0 1 1 1 0 0 0 0 0 0 1 1 0 1)
This decision has five MCexplanations of cardinality 3, e.g.:
(0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0)
The MCexplanation tells us that this Congressmen could have reversed four of their yesvotes, and the classifier would still predict that this Congressman was a Republican.
For a problem of this size, we can enumerate all instances of the classifier. We computed the MCexplanations for each of the positive instances, out of a possible number of instances. Among these MCexplanations, the one that appeared the most frequently was the MCexplanation from the above example. This explanation corresponded to yesvotes on three issues: , , and . Further examination of the dataset revealed that these issues were the three with the fewest Republican novotes.
4 Prime Implicant Explanations
We now consider the second type of explanations, called primeimplicant explanations or PIexplanations for short.
Let and be instantiations of some features and call them partial instances. We will write to mean that extends , that is, it includes but may set some additional features.
Definition 5 (PIExplanation)
Let be a given decision function. A PIexplanation of a decision is a partial instance such that

,

for every , and

no other partial instance satisfies (a) and (b).
Intuitively, a PIexplanation of decision is a minimal subset of instance that makes features outside irrelevant to the decision. That is, we can toggle any feature that does not appear in while maintaining the current decision. The number of features appearing in a PIexplanation will be called the length of the explanation. As we shall see later, PIexplanations of the same decision may have different lengths.
PIexplanations  

        0.0002    () () () () () 
      +  0.0426    () () 
    +    0.0006    () () () 
    +  +  0.1438    () 
  +      0.0016    () () () 
  +    +  0.2933    () 
  +  +    0.0060    () 
  +  +  +  0.6105  +  () 
+        0.0354    () () 
+      +  0.9057  +  () 
+    +    0.1218    () 
+    +  +  0.9732  +  () 
+  +      0.2552    () 
+  +    +  0.9890  +  () 
+  +  +    0.5642  +  () 
+  +  +  +  0.9971  +  () () () 
Table 2 depicts the decision function for the admissions classifier, with PIexplanations for all 16 instances. We write for and () for .
Consider a student (+ +  ) who was not admitted by this decision function. There is a single PIexplanation () for this decision. According to this explanation, it is sufficient to have a poor entrance exam and a poor GPA to be rejected—it does not matter whether they have work experience or if they are a firsttime applicant. That is, we can set these features to any value, and the applicant would still be rejected.
Consider now a student (+ + + +) who was admitted. There are three PIexplanations for this decision, () () (), with different lengths. These explanations can be visualized as (+ * * +), (+ + + *) and (* + + +). This is in contrast to the single MCexplanation (+   +) obtained previously.
4.1 Computing Prime Implicant Explanations
Algorithms exist for converting an OBDD for function into an ODD that encodes the prime implicants of [?; ?; ?].^{6}^{6}6These algorithms compute primeimplicant covers. The resulting ODD has three values for each variable: , and (don’t care). The ODD encodes partial instances, which correspond to the PIexplanations of positive instances (to get the PIexplanations of negative instances, we complement the OBDD ). These algorithms recurse on the structure of the input OBDD, computing prime implicants of subOBDDs. If is the variable labeling the root of OBDD , then denotes its child and denotes its child. Algorithm 4 computes prime implicants by recursively computing prime implicants for , and [?].
As we are interested in explaining a specific instance , we only need the prime implicants compatible with (a function may have exponentially many prime implicants, but those compatible with an instance may be small). We exploit this observation in Algorithm 5, which computes the PIexplanations of a given positive instance by avoiding certain recursive calls. Empirically, we have observed that Algorithm 5 can be twice as fast as Algorithm 4 (computing PIs first, then conjoining with a given instance to obtain PIexplanations). It can also generate ODDs that are an orderofmagnitude smaller. The following table highlights this difference in size and running time, per instance, between Algorithms 4 (cover) & 5 (inst). Relative improvements are denoted by impr; denotes the number of features. We report averages over instances.
time (s)  ODD size  

dataset  cover  inst  impr  cover  inst  impr  
16  0.04  0.02  1.99  2,144  139  15.42  
22  0.06  0.02  2.27  3,130  437  7.14  
16  0.07  0.02  2.56  5,086  446  11.39  
15  0.03  0.02  1.39  432  111  3.89 
4.2 Case Study: Votes Classifier
Consider again the voting record of the Republican Congressman that we considered earlier in Section 3.2:
(0 1 0 1 1 1 0 0 0 0 0 0 1 1 0 1)
There are 30 PIexplanations of this decision. There are 2 shortest explanations of features:
( 0 1 1 0 0 0 1 1 0 )
( 0 1 1 1 0 0 1 1 0 )
The first corresponds to yes votes on:
, , , ,
and no votes on
, , , , .
These 9 votes necessitate the classification of a Republican; no other vote changes this decision. Finally, there are PIexplanations for all decisions made by this classifier:
length of explanation  9  10  11  12  13  total 

number of explanations  35  308  143  19  1  506 
5 More On Monotone Classifiers
We now discuss a specific relationship between MC and PI explanations for monotone classifiers.
An MCexplanation sets all features, while a PIexplanation sets only a subset of the features. For a positive instance, we will say that MCexplanation and PIexplanation match iff can be obtained from by setting all missing features negatively. For a negative instance, MCexplanation and PIexplanation match iff can be obtained from by setting all missing features positively.
Theorem 4
For a decision of a monotone decision function :

Each MCexplanation matches some shortest PIexplanation.

Each shortest PIexplanation matches some MCexplanation.
Hence, for monotone decision functions, MCexplanations coincide with shortest PIexplanations.
The admissions classifier we considered earlier is monotone, which can be verified by inspecting its decision function (in contrast, the votes classifier is not monotone). Here, all MCexplanations matched PIexplanations. For example, the MCexplanation (+   +) for instance (+ +  +) matches the PIexplanation . However, the PIexplanation for instance (+ + + +) does not match the single MCexplanation (+   +). One can verify though, by examining Tables 1 and 2, that shortest PIexplanations coincide with MCexplanations.
MCexplanations are no longer than PIexplanations and their count is no larger than the count of PIexplanations. Moreover, MCexplanations can be computed in linear time, given that the decision function is represented as an OBDD. This is not guaranteed for PIexplanations.
PIexplanations can be directly extended to classifiers with multivalued features. They are also meaningful for arbitrary classifiers, not just monotone ones. While our definition of MCexplanations was directed towards monotone classifiers with binary features, it can be generalized so it remains useful for arbitrary classifiers with multivalued features. In particular, let us partition the values of each feature into two sets: onvalues and offvalues. Let us also partition the set of features into and . Consider now the following question about a decision , where . Keeping fixed, find a culprit of onfeatures in that maintains the current decision. Definition 3 is a special case of this more general definition, and Algorithm 3 can be easily extended to compute these more general MCexplanations using the same complexity (that is, linear in the size of ODD for the decision function).
6 Related Work
There has been significant interest recently in providing explanations for classifiers; see, e.g., [?; ?; ?; ?; ?]. In particular, modelagnostic explainers were sought [?], which can explain the behavior of (most) any classifier, by treating it as a black box. Take for example, LIME, which locally explains the classification of a given instance. Roughly, LIME samples new instances that are “close” to a given instance, and then learns a simpler, interpretable model from the sampled data. For example, suppose a classifier rejects a loan to an applicant; one could learn a decision tree for other instances similar to the applicant, to understand why the original decision was made.
More related to our work is the notion of an “anchor” introduced in [?; ?]. An anchor for an instance is a subset of the instance that is highly likely to be classified with the same label, no matter how the missing features are filled in (according to some distribution). An anchor can be viewed as a probabilistic extension of a PIexplanation. Anchors can also be understood using the SameDecision Probability (SDP) [?; ?; ?], proposed in [?]. In this context, the SDP asks, “Given that I have already observed , what is the probability that I will make the same classification if I observe the remaining features?” In this case, we expect an anchor to have a high SDP, but a PIexplanation will always have an SDP of 1.0.
7 Conclusion
We proposed an algorithm for compiling latenttree Bayesian network classifiers into decision functions in the form of ODDs. We also proposed two approaches for explaining the decision that a Bayesian network classifier makes on a given instance, which apply more generally to any decision function in symbolic form. One approach is based on MCexplanations, which minimize the number of positive features in an instance, while maintaining its classification. The other approach is based on PIexplanations, which identify a smallest set of features in an instance that renders the remaining features irrelevant to a classification. We proposed algorithms for computing these explanations when the decision function has a symbolic and tractable form. We also discussed monotone classifiers and showed that MCexplanations and PIexplanations coincide for this class of classifiers.
Acknowledgments
This work has been partially supported by NSF grant #IIS1514253, ONR grant #N000141512339 and DARPA XAI grant #N660011724032.
Appendix A Proofs

Our proof is based on analyzing Algorithm 2 on an arbitrary latenttree classifier with variables and values, and bounding the size of the decision graph after each call to expandthenmerge. For any iteration of the whileloop, let be the initial decision graph and let be the decision graph generated after the expanding phase of . Furthermore, let denote the number of leaf nodes of (similarly for ). We will show the following loop invariant: . For any iteration, is bounded by , where denotes the depth of . There are variables remaining, and the choice of in the algorithm guarantees that the number of variables under is at most . Thus, is bounded by . If , then . Otherwise if then . Thus, after every call to expandthenmerge, the decision graph has at most leaf nodes and the merging phase cannot increase the number of nodes, giving us a total size bound of . To obtain the size bound of , observe that is at least half of the number of newly expanded nodes for each call, and at most one such call can have nodes. Finally, merging a node in takes time logarithmic in the size of , so the time complexity is .

Our proof is based on [?], which showed that computing the samedecision probability (SDP) is NPhard in naive Bayes networks. Say we have an instance of the number partitioning problem, where we have positive integers and we ask if there exists a set such that . Suppose we have a naive Bayes classifier with features where:
and where we have a uniform prior . Let be the instance where is set to true if and is set to false if . Consider the logodds :
If is a number partitioning solution, then Otherwise where . Hence, if there is no solution , then half of the instances have logodds strictly greater than zero, and the other half have logodds strictly less than zero. Thus, there exists a solution iff the number of positive instances in the decision function of is strictly less than given a (strict) threshold of . Finally, if we can compile the decision function of to an OBDD in polytime, then we can perform model counting in time linear in the size of the OBDD, and hence solve number partitioning, which is NPcomplete. Thus, compiling the decision function is NPhard.

An OBDD can be complemented by simply switching its sink and sink. Since is a conjunction of literals, we can conjoin with by manipulating the OBDD structure directly: if appears in positively (negatively), we redirect the 0edge (1edge) of each OBDD node labeled by to the sink. Clearly, this operation takes time linear in the size of . The operation of minimization can also be performed in time linear in the size of using the technique given in [?] for DNNFs. The minimization procedure performs two passes. The first pass performs an addition or minimization at each node. The second pass redirects some edges depending on simple tests.

Suppose, without loss of generality, that we are explaining a positive instance of a monotone decision function (the negative case is symmetric). The proof uses the following observation: A shortest PIexplanation must have all its features set positively (otherwise, due to monotonicity, we can just drop the negative features in to obtain a shorter PIexplanation).

Suppose that is an MCexplanation. Let be the portion of containing all features that are set positively. Due to monotonicity, we can toggle features of that are outside without changing the decision. Moreover, no subset of will have this property; otherwise, cannot be an MCexplanation. Hence, is a PIexplanation that matches . Suppose now that is not a shortest PIexplanation and let be a shortest PIexplanation. Then we can augment by setting all missing features negatively, giving us a positive instance with a 1cardinality less than that of . Hence, cannot be an MCexplanation.

Suppose that is a shortest PIexplanation. Then all features in must be set positively. Now let be the result of augmenting by setting all missing features negatively. Then is a positive instance since is a PIexplanation. Suppose now that is not an MCexplanation, and let be an MCexplanation. Then let be the portion of containing all features that are set positively. By monotonicity, cannot be a shortest PIexplanation since is shorter than yet all of its completions would be positive instances.

References
 [Bache and Lichman, 2013] K. Bache and M. Lichman. UCI machine learning repository, 2013.
 [Bryant, 1986] R. E. Bryant. Graphbased algorithms for Boolean function manipulation. IEEE Transactions on Computers, C35:677–691, 1986.
 [Chan and Darwiche, 2003] Hei Chan and Adnan Darwiche. Reasoning about Bayesian network classifiers. In Proceedings of the Nineteenth Conference on Uncertainty in Artificial Intelligence (UAI), pages 107–115, 2003.
 [Chen et al., 2014] Suming Chen, Arthur Choi, and Adnan Darwiche. Algorithms and applications for the samedecision probability. Journal of Artificial Intelligence Research, 49:601–633, 2014.
 [Choi et al., 2012] Arthur Choi, Yexiang Xue, and Adnan Darwiche. Samedecision probability: A confidence measure for thresholdbased decisions. International Journal of Approximate Reasoning (IJAR), 53(9):1415–1428, 2012.
 [Choi et al., 2013] Arthur Choi, Doga Kisa, and Adnan Darwiche. Compiling probabilistic graphical models using sentential decision diagrams. In Proceedings of the 12th European Conference on Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU), pages 121–132, 2013.
 [Choi et al., 2017] YooJung Choi, Adnan Darwiche, and Guy Van den Broeck. Optimal feature selection for decision robustness in Bayesian networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), August 2017.
 [Coudert and Madre, 1993] Olivier Coudert and Jean Christophe Madre. Fault tree analysis: prime implicants and beyond. In Proc. of the Annual Reliability and Maintainability Symposium, 1993.
 [Coudert et al., 1993] Olivier Coudert, Jean Christophe Madre, Henri Fraisse, and Herve Touati. Implicit prime cover computation: An overview. In Proceedings of the 4th SASIMI Workshop, 1993.
 [Darwiche and Choi, 2010] Adnan Darwiche and Arthur Choi. Samedecision probability: A confidence measure for thresholdbased decisions under noisy sensors. In Proceedings of the Fifth European Workshop on Probabilistic Graphical Models (PGM), pages 113–120, 2010.
 [Darwiche and Marquis, 2002] Adnan Darwiche and Pierre Marquis. A knowledge compilation map. JAIR, 17:229–264, 2002.
 [Darwiche, 2001] Adnan Darwiche. Decomposable negation normal form. Journal of the ACM, 48(4):608–647, 2001.
 [Elenberg et al., 2017] Ethan R. Elenberg, Alexandros G. Dimakis, Moran Feldman, and Amin Karbasi. Streaming weak submodularity: Interpreting neural networks on the fly. In Advances in Neural Information Processing Systems 30 (NIPS), pages 4047–4057, 2017.
 [Horiyama and Ibaraki, 2002] Takashi Horiyama and Toshihide Ibaraki. Ordered binary decision diagrams as knowledgebases. Artificial Intelligence (AIJ), 136(2):189–213, 2002.
 [Lundberg and Lee, 2017] Scott M. Lundberg and SuIn Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (NIPS), pages 4768–4777, 2017.
 [Meinel and Theobald, 1998] Christoph Meinel and Thorsten Theobald. Algorithms and Data Structures in VLSI Design: OBDD — Foundations and Applications. Springer, 1998.
 [Minato, 1993] Shinichi Minato. Fast generation of primeirredundant covers from binary decision diagrams. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 76(6):967–973, 1993.
 [Ribeiro et al., 2016a] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Nothing else matters: Modelagnostic explanations by identifying prediction invariance. In NIPS Workshop on Interpretable Machine Learning in Complex Systems, 2016.
 [Ribeiro et al., 2016b] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should i trust you?”: Explaining the predictions of any classifier. In Knowledge Discovery and Data Mining (KDD), 2016.
 [Ribeiro et al., 2018] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: Highprecision modelagnostic explanations. In Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence (AAAI), 2018.
 [van der Gaag et al., 2004] Linda C. van der Gaag, Hans L. Bodlaender, and A. J. Feelders. Monotonicity in Bayesian networks. In Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence (UAI), pages 569–576, 2004.
 [Wegener, 2000] Ingo Wegener. Branching Programs and Binary Decision Diagrams. SIAM, 2000.