# Representer Point Selection for

Explaining Deep Neural Networks

###### Abstract

We propose to explain the predictions of a deep neural network, by pointing to the set of what we call representer points in the training set, for a given test point prediction. Specifically, we show that we can decompose the pre-activation prediction of a neural network into a linear combination of activations of training points, with the weights corresponding to what we call representer values, which thus capture the importance of that training point on the learned parameters of the network. But it provides a deeper understanding of the network than simply training point influence: with positive representer values corresponding to excitatory training points, and negative values corresponding to inhibitory points, which as we show provides considerably more insight. Our method is also much more scalable, allowing for real-time feedback in a manner not feasible with influence functions.

Representer Point Selection for

Explaining Deep Neural Networks

Chih-Kuan Yeh^{†}^{†}thanks: Equal contribution Joon Sik Kim ^{1}^{1}footnotemark: 1 Ian E.H. Yen Pradeep Ravikumar
Machine Learning Department
Carnegie Mellon University
Pittsburgh, PA 15213
{cjyeh, joonsikk, eyan, pradeepr}@cs.cmu.edu

noticebox[b]32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.\end@float

## 1 Introduction

As machine learning systems start to be more widely used, we are starting to care not just about the accuracy and speed of the predictions, but also why it made its specific predictions. While we need not always care about the why of a complex system in order to trust it, especially if we observe that the system has high accuracy, such trust typically hinges on the belief that some other expert has a richer understanding of the system. For instance, while we might not know exactly how planes fly in the air, we trust some experts do. In the case of machine learning models however, even machine learning experts do not have a clear understanding of why say a deep neural network makes a particular prediction. Our work proposes to address this gap by focusing on improving the understanding of experts, in addition to lay users. In particular, expert users could then use these explanations to further fine-tune the system (e.g. dataset/model debugging), as well as suggest different approaches for model training, so that it achieves a better performance.

Our key approach to do so is via a representer theorem for deep neural networks, which might be of independent interest even outside the context of explainable ML. We show that we can decompose the pre-activation prediction values into a linear combination of training point activations, with the weights corresponding to what we call representer values, which can be used to measure the importance of each training point has on the learned parameter of the model. Using these representer values, we select representer points – training points that have large/small representer values – that could aid the understanding of the model’s prediction.

Such representer points provide a richer understanding of the deep neural network than other approaches that provide influential training points, in part because of the meta-explanation underlying our explanation: a positive representer value indicates that a similarity to that training point is excitatory, while a negative representer value indicates that a similarity to that training point is inhibitory, to the prediction at the given test point. It is in these inhibitory training points where our approach provides considerably more insight compared to other approaches: specifically, what would cause the model to not make a particular prediction? In one of our examples, we see that the model makes an error in labeling an antelope as a deer. Looking at its most inhibitory training points, we see that the dataset is rife with training images where there are antelopes in the image, but also some other animals, and the image is labeled with the other animal. These thus contribute to inhibitory effects of small antelopes with other big objects: an insight that as machine learning experts, we found deeply useful, and which is difficult to obtain via other explanatory approaches. We demonstrate the utility of our class of representer point explanations through a range of theoretical and empirical investigations.

## 2 Related Work

There are two main classes of approaches to explain the prediction of a model. The first class of approaches point to important input features. Ribeiro et al. [1] provide such feature-based explanations that are model-agnostic; explaining the decision locally around a test instance by fitting a local linear model in the region. Ribeiro et al. [2] introduce Anchors, which are locally sufficient conditions of features that “holds down” the prediction so that it does not change in a local neighborhood. Such feature based explanations are particularly natural in computer vision tasks, since it enables visualizing the regions of the input pixel space that causes the classifier to make certain predictions. There are numerous works along this line, particularly focusing on gradient-based methods that provide saliency maps in the pixel space [3, 4, 5, 6].

The second class of approaches are sample-based, and they identify training samples that have the most influence on the model’s prediction on a test point. Among model-agnostic sample-based explanations are prototype selection methods [7, 8] that provide a set of “representative” samples chosen from the data set. Kim et al. [9] provide criticism alongside prototypes to explain what are not captured by prototypes. Usually such prototype and criticism selection is model-agnostic and used to accelerate the training for classifications. Model-aware sample-based explanation identify influential training samples which are the most helpful for reducing the objective loss or making the prediction. Recently, Koh and Liang [10] provide tractable approximations of influence functions that characterize the influence of each sample in terms of change in the loss. Anirudh et al. [11] propose a generic approach to influential sample selection via a graph constructed using the samples.

Our approach is based on a representer theorem for deep neural network predictions. Representer theorems [12] in machine learning contexts have focused on non-parametric regression, specifically in reproducing kernel Hilbert spaces (RKHS), and which loosely state that under certain conditions the minimizer of a loss functional over a RKHS can be expressed as a linear combination of kernel evaluations at training points. There have been recent efforts at leveraging such insights to compositional contexts [13, 14], though these largely focus on connections to non-parametric estimation. Bohn et al. [13] extend the representer theorem to compositions of kernels, while Unser [14] draws connections between deep neural networks to such deep kernel estimation, specifically deep spline estimation. In our work, we consider the much simpler problem of explaining pre-activation neural network predictions in terms of activations of training points, which while less illuminating from a non-parametric estimation standpoint, is arguably much more explanatory, and useful from an explainable ML standpoint.

## 3 Representer Point Framework

Consider a classification problem, of learning a mapping from an input space (e.g., images) to an output space (e.g., labels), given training points , and corresponding labels . We consider a neural network as our prediction model, which takes the form , where and is the last intermediate layer feature in the neural network for input . Note that is the number of classes, is the dimension of the feature, is a matrix , and is all the parameters to generate the last intermediate layer from the input . Thus are all the parameters of our neural network model. The parameterization above connotes splitting of the model as a feature model and a prediction network with parameters . Note that the feature model can be arbitrarily deep, or simply the identity function, so our setup above is applicable to general feed-forward networks.

Our goal is to understand to what extent does one particular training point affect the prediction of a test point as well as the learned weight parameter . Let be the loss, and be the empirical risk. To indicate the form of a representer theorem, suppose we solve for the optimal parameters for some non-decreasing . We would then like our pre-activation predictions to have the decomposition: . Given such a representer theorem, can be seen as the contribution of the training data on the testing prediction . However, such representer theorems have only been developed for non-parametric predictors, specifically where lies in a reproducing kernel Hilbert space. Moreover, unlike the typical RKHS setting, finding a global minimum for the empirical risk of a deep network is difficult, if not impossible, to obtain. In the following, we provide a representer theorem that addresses these two points: it holds for deep neural networks, and for any stationary point solution.

###### Theorem 3.1.

Let us denote the neural network prediction function by , where and . Suppose is a stationary point of the optimization problem: , where for some . Then we have the decomposition:

where and , which we call a representer value for given .

###### Proof.

We note that can be seen as the resistance for training example feature towards minimizing the norm of the weight matrix . Therefore, can be used to evaluate the importance of the training data have on . Note that for any class , holds by (2). Moreover, we can observe that for to have a significant value, two conditions must be satisfied: (a) should have a large value, and (b) should have a large value. Therefore, we interpret the pre-activation value as a weighted sum for the feature similarity with the weight . When is close to with a large positive weight , the prediction score for class is increased. On the other hand, when is close to with a large negative weight , the prediction score for class is then decreased.

We can thus interpret the training points with negative representer values as inhibitory points that suppress the activation value, and those with positive representer values as excitatory examples that does the opposite. We demonstrate this notion with examples further in Section 4.2. We note that such excitatory and inhibitory points provide a richer understanding of the behavior of the neural network: it provides insight both as to why the neural network prefers a particular prediction, as well as why it does not, which is typically difficult to obtain via other sample-based explanations.

### 3.1 Training an Interpretable Model by Imposing L2 Regularization.

Theorem 3.1 works for any model that performs a linear matrix multiplication before the activation , which is quite general and can be applied on most neural-network-like structures. By simply introducing a L2 regularizer on the weight with a fixed , we can easily decompose the pre-softmax prediction value as some finite linear combinations of a function between the test and train data. We now state our main algorithm. First we solve the following optimization problem:

(3) |

Note that for the representer point selection to work, we would need to achieve a stationary point with high precision. In practice, we find that using a gradient descent solver with line search or LBFGS solver to fine-tune after converging in SGD can achieve highly accurate stationary point. Note that we can perform the fine-tuning step only on , which is usually efficient to compute. We can then decompose by Theorem 3.1 for any arbitrary test point , where is the contribution of training point on the pre-softmax prediction . We emphasize that imposing L2 weight decay is a common practice to avoid overfitting for deep neural networks, which does not sacrifice accuracy while achieving a more interpretable model.

### 3.2 Generating Representer Points for a Given Pre-trained Model.

We are also interested in finding representer points for a given model that has already been trained, potentially without imposing the L2 regularizer. While it is possible to add the L2 regularizer and retrain the model, the retrained model may converge to a different stationary point, and behave differently compared to the given model, in which case we cannot use the resulting representer points as explanations. Accordingly, we learn the parameters while imposing the L2 regularizer, but under the additional constraint that be close to . In this case, our learning objective becomes instead of , and our loss can be written as .

###### Definition 3.1.

We say that a convex loss function is “suitable” to an activation function , if it holds that for any , we have = .

Assume that we are given such a loss function that is “suitable to” the activation function . We can then solve the following optimization problem:

(4) |

The optimization problem can be seen to be convex under the assumptions on the loss function. The parameter controls the trade-off between the closeness of and , and the computational cost. For a small , could be arbitrarily close to , while the convergence time may be long. We note that the learning task in Eq. (4) can be seen as learning from a teacher network and imposing a regularizer to make the student model capable of generating representer points. In practice, we may take as an initialization for and perform a simple line-search gradient descent with respect to in (4). In our experiments, we discover that the training for (4) can converge to a stationary point in a short period of time, as demonstrated in Section 4.5.

We now discuss our design for the loss function that is mentioned in . When is the softmax activation, we choose the softmax cross-entropy loss, which computes the cross entropy between and for . When is ReLU activation, we choose , where is the element-wise product. In the following Proposition, we show that and are convex, and satisfy the desired suitability property in Definition 3.1. The proof is provided in the supplementary material.

###### Proposition 3.1.

The loss functions and are both convex in . Moreover, is “suitable to” the softmax activation, and is “suitable to” the ReLU activation, following Definition 3.1.

As a sanity check, we perform experiments on the CIFAR-10 dataset [15] with a pre-trained VGG-16 network [16]. We first solve (4) with loss for , and then calculate as in (2) for all train and test points. We note that the computation time for the whole procedure only takes less than a minute, given the pre-trained model. We compute the Pearson correlation coefficient between the actual output and the predicted output for multiple points and plot them in Figure 1. The correlation is almost 1 for both train and test data, and most points lie at the both ends of line.

We note that Theorem 3.1 can be applied to any hidden layer with ReLU activation by defining a sub-network from input and the output being the hidden layer of interest. The training could be done in a similar fashion by replacing with . In general, any activation can be used with a derived "suitable loss".

## 4 Experiments

We perform a number of experiments with multiple datasets and evaluate our method’s performance and compare with that of the influence functions.^{1}^{1}1Source code available at github.com/yankeesrules/Representer_Point_Selection. The goal of these experiments is to demonstrate that selecting the representer points is efficient and insightful in several ways. Additional experiments discussing the differences between our method and the influence function are included in the supplementary material.

### 4.1 Dataset Debugging

To evaluate the influence of the samples, we consider a scenario where humans need to inspect the dataset quality to ensure an improvement of the model’s performance in the test data. Real-world data is bound to be noisy, and the bigger the dataset becomes, the more difficult it will be for humans to look for and fix mislabeled data points. It is crucial to know which data points are more important than the others to the model so that prioritizing the inspection can facilitate the debugging process.

To show how well our method does in dataset debugging, we run a simulated experiment on CIFAR-10 dataset [17] with a task of binary classification with logistic regression for the classes automobiles and horses. The dataset is initially corrupted, where 40 percent of the data has the labels flipped, which naturally results in a low test accuracy of . The simulated user will check some fraction of the train data based on the order set by several metrics including ours, and fix the labels. With the corrected version of the dataset, we retrain the model and record the test accuracies for each metrics. For our method, we train an explainable model by mimimizing (3) as explained in section 3.1. The L2 weight decay is set to for all methods for fair comparison. All experiments are repeated for 5 random splits and we report the average result. In Figure 2 we report the results for four different metrics: “ours” picks the points with bigger for training instance and its corresponding label ; “influence” prioritizes the training points with bigger influence function value; and “random” picks random points. We observe that our method recovers the same amount of training data as the influence function while achieving higher testing accuracy. Nevertheless, both methods perform better than the random selection method.

### 4.2 Excitatory (Positive) and Inhibitory (Negative) Examples

We visualize the training points with high representer values (both positive and negative) for some test points in Animals with Attributes (AwA) dataset [18] and compare the results with those of the influence functions. We use a pre-trained Resnet-50 [19] model and fine-tune on the AwA dataset to reach over 90 percent testing accuracy. We then generate representer points as described in section 3.2. For computing the influence functions, just as described in [10], we froze all top layers of the model and trained the last layer. We report top three points for two test points in the following Figures 3 and 4. In Figure 3, which is an image of three grizzly bears, our method correctly returns three images that are in the same class with similar looks, similar to the results from the influence function. The positive examples excite the activation values for a particular class and supports the decision the model is making. For the negative examples, just like the influence functions, our method returns images that look like the test image but are labeled as a different class. In Figure 4, for the image of a rhino the influence function could not recover useful training points, while ours does, including the similar-looking elephants or zebras which might be confused as rhinos, as negatives. The negative examples work as inhibitory examples for the model – they suppress the activation values for a particular class of a given test point because they are in a different class despite their striking similarity to the test image. Such inhibitory points thus provide a richer understanding, even to machine learning experts, of the behavior of deep neural networks, since they explicitly indicate training points that lead the network away from a particular label for the given test point. More examples can be found in the supplementary material.

### 4.3 Understanding Misclassified Examples

The representer values can be used to understand the model’s mistake on a test image. Consider a test image of an antelope predicted as a deer in the left-most panel of Figure 5. Among 181 test images of antelopes, the total number of misclassified instances is 15, among which 12 are misclassified as deer. All of those 12 test images of antelopes had the four training images shown in Figure 5 among the top inhibitory examples. Notice that we can spot antelopes even in the images labeled as zebra or elephant. Such noise in the labels of the training data confuses the model – while the model sees elephant and antelope, the label forces the model to focus on just the elephant. The model thus learns to inhibit the antelope class given an image with small antelopes and other large objects. This insight suggests for instance that we use multi-label prediction to train the network, or perhaps clean the dataset to remove such training examples that would be confusing to humans as well. Interestingly, the model makes the same mistake (predicting deer instead of antelope) on the second training image shown (third from the left of Figure 5), and this suggests that for the training points, we should expect most of the misclassifications to be deer as well. And indeed, among 863 training images of antelopes, 8 are misclassified, and among them 6 are misclassified as deer.

### 4.4 Sensitivity Map Decomposition

From Theorem 3.1, we have seen that the pre-softmax output of the neural network can be decomposed as the weighted sum of the product of the training point feature and the test point feature, or . If we take the gradient with respect to the test input for both sides, we get . Notice that the LHS is the widely-used notion of sensitivity map (gradient-based attribution), and the RHS suggests that we can decompose this sensitivity map into a weighted sum of sensitivity maps that are native to each -th training point. This gives us insight into how sensitivities of training points contribute to the sensitivity of the given test image.

In Figure 6, we demonstrate two such examples, one from the class zebra and one from the class moose from the AwA dataset. The first column shows the test images whose sensitivity maps we wish to decompose. For each example, in the following columns we show top four influential representer points in the the top row, and visualize the decomposed sensitivity maps in the bottom. We used SmoothGrad [20] to obtain the sensitivity maps.

For the first example of a zebra, the sensitivity map on the test image mainly focuses on the face of the zebra. This means that infinitesimally changing the pixels around the face of the zebra would cause the greatest change in the neuron output. Notice that the focus on the head of the zebra is distinctively the strongest in the fourth representer point (last column) when the training image manifests clearer facial features compared to other training points. For the rest of the training images that are less demonstrative of the facial features, the decomposed sensitivity maps accordingly show relatively higher focus on the background than on the face. For the second example of a moose, a similar trend can be observed – when the training image exhibits more distinctive bodily features of the moose than the background (first, second, third representer points), the decomposed sensitivity map highlights the portion of the moose on the test image more compared to training images with more features of the background (last representer point). This provides critical insight into the contribution of the representer points towards the neuron output that might not be obvious just from looking at the images itself.

### 4.5 Computational Cost and Numerical Instabilities

Computation time is particularly an issue for computing the influence function values [10] for a large dataset, which is very costly to compute for each test point. We randomly selected a subset of test points, and report the comparison of the computation time in Table 1 measured on CIFAR-10 and AwA datasets. We randomly select 50 test points to compute the values for all train data, and recorded the average and standard deviation of computation time. Note that the influence function does not need the fine-tuning step when given a pre-trained model, hence the values being 0, while our method first optimizes for using line-search then computes the representer values. However, note that the fine-tuning step is a one time cost, while the computation time is spent for every testing image we analyze. Our method significantly outperforms the influence function, and such advantage will favor our method when a larger number of data points is involved. In particular, our approach could be used for real-time explanations of test points, which might be difficult with the influence function approach.

Influence Function | Ours | |||
---|---|---|---|---|

Dataset | Fine-tuning | Computation | Fine-tuning | Computation |

CIFAR-10 | 0 | |||

AwA | 0 |

While ranking the training points according to their influence function values, we have observed numerical instabilities, more discussed in the supplementary material. For CIFAR-10, over 30 percent of the test images had all zero training point influences, so influence function was unable to provide positive or negative influential examples. The distribution of the values is demonstrated in Figure 7, where we plot the histogram of the maximum of the absolute values for each test point in CIFAR-10. Notice that over 300 testing points out of 1,000 lie in the first bin for the influence functions (right). We checked that all data in the first bin had the exact value of 0. Roughly more than 200 points lie in range , the values which may create numerical instabilities in computations. On the other hand, our method (left) returns non-trivial and more numerically stable values across all test points.

## 5 Conclusion and Discussion

In this work we proposed a novel method of selecting representer points, the training examples that are influential to the model’s prediction. To do so we introduced the modified representer theorem that could be generalized to most deep neural networks, which allows us to linearly decompose the prediction (activation) value into a sum of representer values. The optimization procedure for learning these representer values is tractable and efficient, especially when compared against the influence functions proposed in [10]. We have demonstrated our method’s advantages and performances on several large-scale models and image datasets, along with some insights on how these values allow the users to understand the behaviors of the model.

An interesting direction to take from here would be to use the representer values for data poisoning just like in [10]. Also to truly see if our method is applicable to several domains other than image dataset with different types of neural networks, we plan to extend our method to NLP datasets with recurrent neural networks. The result of a preliminary experiment is included in the supplementary material.

### Acknowledgements

We acknowledge the support of DARPA via FA87501720152, and Zest Finance.

## References

- Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM, 2016.
- Ribeiro et al. [2018] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. 2018.
- Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Shrikumar et al. [2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685, 2017.
- Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365, 2017.
- Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
- Bien and Tibshirani [2011] Jacob Bien and Robert Tibshirani. Prototype selection for interpretable classification. The Annals of Applied Statistics, pages 2403–2424, 2011.
- Kim et al. [2014] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems, pages 1952–1960, 2014.
- Kim et al. [2016] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems, pages 2280–2288, 2016.
- Koh and Liang [2017] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885–1894, 2017.
- Anirudh et al. [2017] Rushil Anirudh, Jayaraman J Thiagarajan, Rahul Sridhar, and Timo Bremer. Influential sample selection: A graph signal processing approach. arXiv preprint arXiv:1711.05407, 2017.
- Schölkopf et al. [2001] Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory, pages 416–426. Springer, 2001.
- Bohn et al. [2017] Bastian Bohn, Michael Griebel, and Christian Rieger. A representer theorem for deep kernel learning. arXiv preprint arXiv:1709.10441, 2017.
- Unser [2018] Michael Unser. A representer theorem for deep neural networks. arXiv preprint arXiv:1802.09210, 2018.
- Krizhevsky and Hinton [2009] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
- Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Xian et al. [2017] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. arXiv preprint arXiv:1707.00600, 2017.
- He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- Smilkov et al. [2017] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
- Maas et al. [2011] Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142–150. Association for Computational Linguistics, 2011.

## Appendix A Proof of Proposition 3.1

###### Proof.

The convexity can be checked easily. If , by the first order condition we have

(5) |

By chain rule we obtain

(6) |

and thus we have

(7) |

When is not a zero vector, this reduces to

(8) |

and we show that . As a result, is “suitable to” the softmax activation.

If , we can rewrite as

(9) |

and we note that , the -th row for , is only related to . Therefore, we have . We now consider the cases where and .

When ,

(10) |

obtains the minimum when , therefore .

When ,

For , the minimum for is 0. For , the minimum for is only if . Therefore, the minimum is reached again when . As a result, is “suitable to” the ReLU activation. ∎

## Appendix B Relationship with the Influence Function

In this section, we compare the behaviors of our method and the influence functions [10]. Recall that for a training point and a test point , the influence function value is computed in terms of the gradients/hessians of the loss, and it reflects how the loss at a test point will change when the training point is perturbed (weighted more/less). Because the influence function is defined in terms of the loss, its value can easily become arbitrarily close to zero if the the loss is flat in some region. On the other hand, our representer values are computed using the neurons’ activation values, which may result in comparatively larger values in general. We verify this in a toy dataset in 2-D with a large margin shown in Figure 8.

We train a multi-layer perceptron with ReLU activations as a binary classifier. The influential points, we would expect, are the points that are closer to the decision boundary. As shown on the left of Figure 8, the influence function does not provide positive or negative examples from each class for the given test point in green square because all training points have exactly zero influence function values, marked with cyan crosses. However, our method provides correct positive and negative points near the decision boundary as we can see from the rightmost panel of Figure 8.

In Figure 9, we compare the behaviors of several metrics (Euclidean distance, influence function, representer value) for selecting training points that are most similar to a test point from CIFAR-10 dataset. We use a pre-trained VGG-16. As we can observe from the first two plots, the Euclidean distance does not reflect the class each training point is in – even if the training point is far away from the test point it may still be a similar image in the same class. On the other hand, representer and influence function values tell us about which class each training point is in. From the third plot, we observe that influence function and representer values agree on selecting images of horses as harmful and dogs as helpful.

## Appendix C More Examples of Positive/Negative Reperesenter Points

More examples of positive and negative representer points are shown in Figure 10. Observe that the positive points all have the same class label as the test image with high resemblance, while the negative points, despite their similarity to the test image, have different classes.

## Appendix D Representer Points of LSTM on NLP Data

We perform a preliminary experiment on an LSTM network trained on a IMDB movie review dataset [21]. Each data point is a review about a movie, and the task is to identify whether the review has a positive or negative sentiment. The pretrain model achieves accuracy. We obtain the positive and negative repesenter points with methods described in section 3.2. In Table 2, we show the test review which is predicted as a negative review by the LSTM network. We observe that both the top-1 positive and negative representer contain negative connotations just like the test review, but in the negative representer (which is a positive review about the movie) the negative connotation comes from a character in the movie rather than from a reviewer.

Test Review | <START> when i first saw this movie in the theater i was so angry it completely |

blew in my opinion i didn’t see it for a decade then decided what the hell let’s see | |

i’m watching all <> movies now to see where it went wrong my guess is it was with | |

sequel 5 that was the first to <> the whole i am in a dream <> i see weird stuff oh <> | |

it’s not a dream oh wait i see something spooky oh never mind <> storyline those | |

which made it so scary in the first place nothing fantasy nothing weird the box got | |

opened boom they came was the only one that could bargain her way out of it first | |

Positive Representer | <START> no not the <> of <> the <> <> <> but the mini series <> <> lifetime must |

have realized what a dog this was because the series was burned off two episodes at a | |

time most of them broadcast between 11 p m friday nights and 1 a m saturday <> as to | |

why i watched the whole thing i can only <> to <> sudden <> attacks of <> br br most | |

of the cast are <> who are likely to remain unknown the only two <> names are shirley | |

jones and rachel ward who turn in the only decent performances jones doesn’t make it | |

through the entire series lucky woman ward by the way is aging quite well since her | |

Negative Representer | <START> this time around <> is no longer royal or even particularly close to being any |

such thing instead rather a butler to the prince <> portrayed by hugh <> who <> tim <> | |

who presence is <> missed and that hole is never filled his character had an innocent charm | |

was a bumbling and complete <> we can’t help but care for him which isn’t at all true of his | |

<> as being <> which he apparently was according to the <> page not to mention loud | |

<> and utterly non threatening <> can now do just about what he <> and does so why is | |

he so frustrated and angry honestly it gets depressing at times yes his master is a <> they |