Improved Explainability of Capsule Networks: Relevance Path by Agreement
Abstract
Recent advancements in signal processing and machine learning domains have resulted in an extensive surge of interest in deep learning models due to their unprecedented performance and high accuracy for different and challenging problems of significant engineering importance. However, when such deep learning architectures are utilized for making critical decisions such as the ones that involve human lives (e.g., in medical applications), it is of paramount importance to understand, trust, and in one word “explain” the rational behind deep models’ decisions. Currently, deep learning models are typically considered as blackbox systems, which do not provide any clue on their internal processing actions. Although some recent efforts have been initiated to explain behavior and decisions of deep networks, explainable artificial intelligence (XAI) domain is still in its infancy. In this regard, we consider capsule networks (referred to as CapsNets), which are novel deep structures; recently proposed as an alternative counterpart to convolutional neural networks (CNNs), and posed to change the future of machine intelligence. In this paper, we investigate and analyze structures and behaviors of the CapsNets and illustrate potential explainability properties of such networks. Furthermore, we show possibility of transforming deep learning architectures in to transparent networks via incorporation of capsules in different layers instead of convolution layers of the CNNs.
Improved Explainability of Capsule Networks: Relevance Path by Agreement
Atefeh Shahroudnejad, Arash Mohammadi, and Konstantinos N. Plataniotis 
Concordia Institute for Information Systems Engineering, Concordia University, Montreal, QC, Canada 
Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON, Canada 
Emails: a hahrou, arashmoh@encs.concordia.ca; kostas@ece.utoronto.ca ^{†}^{†}thanks: This work was partially supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada through the NSERC Discovery Grant RGPIN2016049988. 
Index Terms: Explainable Machine Learning, Capsule Networks, Deep Neural Networks, Convolutional Neural Networks.
1 Introduction
Nowadays, advanced machine learning techniques [1] have encompassed all aspects of human life including complicated tasks. As such, several critical decisions are now made based on predictions provided by machine learning models without any human supervision or participation. It is, therefore, of paramount importance to be able to trust a model, validate its predictions, and make sure that it performs completely well on unseen or unfamiliar real world data. For example, in critical domains such as medical applications [2] or selfdriving cars [3], even a single incorrect decision is not acceptable and could possibly lead to a catastrophic result. For guaranteeing reliability of a machine learning model, it is significantly important to understand and analyze rational reasons behind the decisions made by such sophisticated and advanced models, in other words, we need to be able to open the blackbox. On the other hand, deep models [4] are considered as one of the most successful methods in several areas especially in image processing and computer vision domains. However, as deep architectures become more complex and introduce more nonlinearity, their structures become less transparent and it is harder to understand what operations or input information lead to a specific decision. Moreover, in scenarios where sufficient data for training is not available, which is a common case when it comes to training deep neural networks such as Convolutional Neural Nets (CNNs) [5], probability of error will increase drastically, which further necessitates an urgent need for interpretation of machine learning architectures.
Explaining a model means providing extra qualitative information about why the model reach to a specific decision regarding the input components’ relationship (e.g., different patches of an image [6]). Explanation can be considered as opening incomprehensible blackbox and seeing inside. In other words, the main goal is to find answers to questions of like: What is happening inside a neural network? What does each layer of a deep architecture do? What features a deep network is looking for? Fig. 1 illustrates a graphical representation of this blackbox concept, where the input is an image and the network prediction would be a single word (e.g., face or cat). As can be seen, such a single output provides no evidence for confirming the truth of predictions or rejecting incorrect predictions without having access to the groundtruth. Main advantages of using an explainable model are as follows: Verification of the model; Improving it by understanding failure points; Extracting new insights and hidden laws of the model, and; Finally, identifying modules responsible for incorrect decisions [7].
Explainability can be fulfilled visually, text based, example based, and/or by relating inputs and learned parameters [8]. In this regard, recently there has been a great surge of interest [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] for development of methodologies to make neural network models explainable. For example, an early work in this area is named sensitivity analysis (SA) [10], which measures how much changing each pixel effects the final decision and then a heatmap is obtained referred to as the explainability feature. But this heatmap does not show which patches or pixels play more important role in the decision making process. Moreover, it more looks like a saliency map [17], which uses unique frequencies or focus points or other features to find regions of interest, however, pixels identified as salient regions are not necessarily the pixels being involved in making the predictions. In layerwise relevance propagation method (LRP) [11], each prediction is decomposed by redistributing backward through the network’s layers using redistribution rules and finding a relevance path. Relevance of each neuron or pixel indicates how much it contributes to the decision. The LRP approach is unsatisfying when we have a more complex or nested architecture. Reference [6], proposed Local Interpretable Modelagnostic Explanations (LIME) approach, which explains the prediction by approximating the original model with an interpretable model around several local neighbourhoods. Reference [12] proposed to identify the most responsible training points by tracing back the prediction using influence functions, which estimate effects of changing each training point on that prediction. Gradientweighted Class Activation Mapping (GradCAM) [18] approach is another explanation method for CNNs which uses gradient to obtain localization map as a visual explanation and finds important layers for each class. Finally, Reference [19] proposed a verification framework for increasing trustworthy and explainability of deep networks. It assumes that there is a region in each input which specifically determines its class category and if the prediction shows this category, its input should include the saliency points of that region.
Although recently different research works are developed for explaining complex behaviour of deep neural networks, especially for visual tasks as briefly outlined above, explainable artificial intelligence (XAI) is still in its infancy and needs significant research to further open the blackbox. In this regard, we focus on a very recently proposed deep network architecture referred to as Capsule Networks (CapsNets) [20], which is a turning point in the deep learning research. In particular, we investigate CapsNets’ architecture and behavior from explanability perspective. Through analyzing different underlying components of the CapsNet architecture and its learning mechanism (routingbyagreement), we illustrate that this revolutionized deep net has more intrinsic explainability properties. In particular, we show that CapsNets automatically form the verification framework introduced in [19], learn regions of interest which determine class category, and as such improve trustworthy and explainability of deep networks. Besides, CapsNets inherently create the relevancy path (introduced in [6]) and we refer to it as relevance by agreement as a biproduct of the routingbyagreement algorithm, which adds another level of explanability to this architecture.
2 CapsNets Problem Formulation
Generally speaking, CNNs learn to classify objects using convolutional operations by first extracting low level features in initial layers and then stacking up the preliminary features learned through initial layers to extract more complex features in higher layers. However, CNN architecture ignores the hierarchy between layers (as is present in the human brain), which limits their modeling capabilities. CNNs try to overcome (mask) such limitations by utilization of a large amount of training data. The first problem (limitation) is that CNNs use subsampling in pooling steps to transfer more important features to the next layer. Therefore, some viewpoint changes and precise spatial relationships between higher level components will be lost. The other problem is that CNNs are not robust against new viewpoints, because they can not extrapolate their geometric information. For these reasons, CapsNet has been proposed to replace invariance concept with equivalence, which means that if the input is spatially transformed, the network adapts accordingly and responses properly. A capsule is a group of neurons nested inside a layer and is considered as the main building block of the CapsNet architecture. Actually, the idea behind introducing a capsule is to encapsulate (possibly large) number of pose information (e.g., position, orientation, scaling, and skewness) together with other instantiation parameters (such as color and texture) for different parts or fragments of an object. This multilayer structure is deep in width instead of being deep in height. It can be considered as a parse tree because each active capsule chooses a capsule in the next layer as its parent in the tree. By incorporation of capsules instead of neurons and deepening the width, therefore, CapsNets can better handle different visual stimulus and provide a better translational invariance compared to pooling methods (max/average pooling) used in CNNs. Fig. 2 shows a nose CapsNet for face detection problem. In this illustrative example, there are 5 possible facial component capsules (we note that these are being automatically formed by the CapsNet) and 4 instantiation parameters (i.e., shifting, scaling, rotation, and deformation), which are also extracted automatically by the network. It is worth mentioning that in designing a CapsNet architecture, we only specify the number of capsules and the number of instantiation parameters per capsule, the network then learns specifics automatically.
Fig. 3 shows detection architecture for a CapsNet with the following three layers: (i) Convolutional layers; (ii) A PrimaryCapsule (PC) layer (the first capsule layer of a CapsNet architecture) consisting of capsules, and; (iii) A ClassCapsule (CC) layer (the last capsule layer of a CapsNet architecture) with capsules. For simplicity of the presentation, first we further describe the three layers CapsNet shown in Fig. 3. In this architecture, first the input is fed into the convolutional layers for extracting local features from the pixels. The next layer is the PC layer, where each capsule , for (), has an activity vector () to encode spatial information (instantiation parameters). Capsules in the PC layer can be grouped into several blocks referred to as “Component Capsules”. In the next step, the output vector of the PC, for (), is fed into the CC, for (), using weight matrix and “Coupling Coefficient” as follows
(1) 
where is the prediction vector indicating how much the PC contributes to the CC , and scalar is a coupling coefficient which links predictions of the PC to the CC . Note that, vector is a weighted sum of all the PC predictions for the CC . Also we note that, denotes the dimension of each capsule in the PC layer while denotes the dimension of each capsule in the CC layer. We also note that, one can introduce number of intermediate capsule layers located between the PC and CC layers each with number of localized capsules. If needed, we use index , for (), to refer to the intermediate capsule layers.
Coupling coefficients are trained during the routing process such that for each capsule in the CC layer, we have . Finally, the vector output for the CC is obtained through nonlinear squashing operator given by
(2) 
In each iteration, coupling coefficient is updated by agreement (using dot product) between vector , which is the output of the CC , and vector , which is the prediction vector of the PC. Length of vector associated with the CC , for (), indicates the presence of an object represented by class in the input.
Let us further elaborate on the CapsNet structure introduced above, in terms of the example shown in Fig. 3. Here, we have dimensional PCs, i.e., each capsule entity in the PC layer encodes eight different types of information. Besides, the PC layer consists of component capsules (cubes) each of which consisting of capsules, shown by “arrows” in the last component capsule. Therefore, the PC layer in this illustrative example has individual capsules. Dimension of capsules in the CC layer is and there are total of individual capsules in the CC layer. Finally, in terms of our running example, has the largest magnitude among the CCs considered here, therefore, the CapsNet’s output decision will be digit 6. This was a brief overview of the CapsNet architecture, next we will investigate its explanability properties.
3 CapsNets’ Explainability
In this section, we look at CapsNet architecture described in the previous section from different angles to see whether or not this revolutionized deep learning model has improved explanability characteristics. As we mentioned previously, CNNs can not preserve spatial relationship between components of an object because, some features will be discarded during the pooling process. CNNs compensate this deficiency by increasing the number of training data. In CapsNets, we have capsules (a group of neurons) instead of neurons (single units). Each capsule performs some internal complicated computations and its output is a vector instead of a scalar value. In particular, we investigate the potential properties of using such vectorized outputs, which could lead to more explainability.
3.1 Relevance Path by Agreement
The vector representation provided by CapsNets is highly informative and can model possible instantiation parameters for components or fragments of an object [21]. We argue that this vector output of the capsules can essentially lead to improved explanability of the overall network. In other words, while the length (magnitude) of the output vector corresponding to capsule in the CC layer is used to make decisions regarding the input image, the length of the output vector from the of capsule in the PC layer or an intermediate capsule layer can be interpreted as probability of existence of the feature that this capsule has been trained to detect. More specifically, we can assign to each capsule a set consisting of two segments for explanation purposes:

Likelihood values which can be used to explain existence probability of the feature that a capsule detects, and;

Instantiation parameter vector values which can be used to explain consistency among the layers. In other words, when all capsules of an object are in an appropriate relationship with consistence parameters, the higher level capsule of that object will have a higher likelihood. Therefore, explanations can be provided to describe why the network did detect an object.
For example, Fig. 4 shows the sets computed based on intermediate component capsules referring to class face () in the CC layer. Regarding Item (i), the likelihood part of each of these capsules is relatively high explaining that the input contains all the facial components represented by these 5 component capsules with high probability. However, the network decision is that there is not a face in the input as the likelihood of the face capsule in the CC layer is relatively low. This can be explained based on the nonconsistency among the instantiation parameters (Item (ii)). The output vector of the capsule in the PC layer is multiplied by its weight matrix corresponding to the capsule in the CC layer, which has been learned through backpropagation algorithm to account for all possible transformations of the instantiation parameters. This multiplication is considered as the vote of capsule for class (which is represented by vector ). When all related capsules have similar votes for the CC, it means that they agree to each other on presence of object .
CapsNet applies nonlinear squashing function on output vectors () in each iteration. It actually bounds likelihood of these vectors between 0 and 1, which means that it suppresses small vectors and preserves long vectors in the unit length
(3) 
Therefore, during agreement iterations, unrelated capsules will become smaller and smaller and the related ones will be remained unchanged. Consequently, introduction of the squashing function results in the coupling coefficients associated with irrelevant capsules to approach zero while coupling coefficient corresponding to the ones responsible for the CC to increase. Hence, CapsNets intrinsically construct a relevance path (we refers to it as the relevance path by agreement concept) which eliminates the need for a backward process to construct the relevance path. The reason behind the exitance of the relevance path by agreement is that CapsNet uses dynamic routing instead of common pooling methods. In the other words, when a group of capsules agree for a parent (higher level component), they construct a part whole relationship which can be considered as a relevance path. For example in face prediction case, facial components (eyes, nose, mouth) in a particular relationship will detect a face as a higher level component (Fig. 2).
4 Experimental Setup
Conv with kernels, stride=1, Relu 
Conv with kernels, stride=2, 8D Caps 
Multiply by , Weighted sum, Squashing 
Decoder part 
In this section, we investigate explanation capabilities of the CapsNets on MNIST dataset [22]. The used architecture is similar to the one that has been presented in [20] as outlined in Table 1. After training CapsNet with MNIST training data, in a first experiment we varied each of the two parameters of the CC output vector corresponding to the detected digit and reconstructed it again to find out what is nature of the two plausible features that have been learned (i.e., to explain the learned features at the CC layer). Fig. 5 illustrates the results for three digits. It is observed that by changing the parameters, thickness and shape of digits have been changed simultaneously. Therefore, we can consider them as explanation of the learned features. Fig. 6 displays the two parameters of the output vector associated with the detected digit capsule for all the testing dataset. As can be seen, the vector output of different digit capsules significantly overlap with each other, e.g., capsules 1/7; 6/9, and; 3/5 generate close values. Therefore, in misclassified samples, there is a high probability for detecting overlapped digits. Recognizing these failure points is another level of explanation that CapsNet can provide on MNIST dataset. Now, we found misclassified samples in 1000 testing data and display their output prediction vectors. In all cases, the true class had the second highest likelihood (magnitude) except one instance in which the third highest likelihood was corresponding to the true label. Moreover, we can usually see a high decreasing magnitude between third and forth positions. Therefore, one can use the likelihood values and present the second and third highest capsules as alternative solutions and explanations. Fig. 7 shows all misclassified samples in 1000 testing data. We reconstructed input digits by prediction and true label outputs. As we see, each input digit in the first row is similar to both reconstructed digits bellow it, which explains why the model has been failed in these cases.
5 Conclusion
In this paper, we represented the necessity of explainability in deep neural networks especially in critical decisions where a single incorrect decision is even unacceptable. Previous explainability methods try to find and visualize the most relevant pixels or neurons by adding an extra explanation phase. In this work, we illustrated potential intrinsic explainability properties of Capsule network, by analyzing its behavior and structure.
References
 [1] C.M. Bishop, “Pattern Recognition and Machine Learning,” SpringerVerlag New York, 2006.
 [2] M. Havaei, A. Davy, D. WardeFarley, A. Biard, A. Courville, Y. Bengio, C. Pal, P.M. Jodoin, and H. Larochelle, “Brain Tumor Segmentation with Deep Neural Networks,” arXiv preprint arXiv:1505.03540, 2015.
 [3] M. Bojarski, D.D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L.D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to End Learning for SelfDriving Cars,” arXiv preprint arXiv:1604.07316, 2016.
 [4] I. Goodfellow, Y. Bengio, and A. Courville, “Deep Learning,” MIT press, 2016.
 [5] A. Krizhevsky, I. Sutskever and G.E .Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” Advances in neural information processing systems, pp. 10971105, 2012.
 [6] M.T. Ribeiro, S. Singh and C. Guestrin, “Why Should I Trust You?: Explaining the Predictions of Any Classifier,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 11351144, 2016.
 [7] W. Samek, T. Wiegand, and K.R. Müller, “Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models,” ITU Journal: ICT Discoveries, Special Issue The Impact of AI on Communication Networks and Services, vol. 1, pp. 110, 2017.
 [8] Z.C. Lipton, “The Mythos of Model Interpretability,” arXiv preprint arXiv:1606.03490, 2016.
 [9] M. Kahng, P.Y. Andrews, A. Kalro, and D.H. Chau, “ActiVis: Visual Exploration of IndustryScale Deep Neural Network Models,” IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 8897, Jan. 2018.
 [10] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, K.R. Müller “How to Explain Individual Classification Decisions,” Journal of Machine Learning Research, vol. 11, pp. 18031831, 2010.
 [11] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.R. Müller, W. Samek, “On PixelWise Explanations for Nonlinear Classifier Decisions by LayerWise Relevance Propagation,” PloS one, vol. 10, no. 7, pp. 130140, 2015.
 [12] P.W. Koh, and P. Liang, “Understanding BlackBox Predictions via Influence Functions,” arXiv preprint arXiv:1703.04730, 2017.
 [13] C.C. Jay Kuo, Yueru Chen, “On datadriven Saak Transform,” Journal of Visual Communication and Image Representation, vol. 50, pp. 237246, 2018
 [14] W. Guo, K. Zhang, L. Lin, S. Huang, X. Xing, “Towards Interrogating Discriminative Machine Learning Models,” arXiv preprint arXiv:1705.08564 , 2017.
 [15] W. Samek, T. Wiegand, and K.R. Müller, “Methods for Interpreting and Understanding Deep Neural Networks,” Digital Signal Processing, vol. 73, pp. 115, 2018.
 [16] W. Samek, A. Binder, G. Montavon, S. Bach,and K.R. Müller, “Evaluating the Visualization of What a Deep Neural Network has Learned,” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 11, pp. 26602673, 2017.
 [17] S. Goferman, L. ZelnikManor, and A. Tal, “Contextaware Saliency Detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 10, pp. 19151926, 2012.
 [18] R.R. Selvaraju, A. Das, R. Vedantam and M. Cogswell, “GradCAM: Why did you say that?” arXiv preprint arXiv:1611.07450, 2016.
 [19] X. Huang, M. Kwiatkowska, S. Wang and M. Wu, “Safety Verification of Deep Neural Networks,” In International Conference on Computer Aided Verification pp. 329. Springer, Cham, 2017.
 [20] S. Sabour, N. Frosst, and G.E. Hinton, “Dynamic Routing between Capsules,” Advances in Neural Information Processing Systems, pp. 38593869. 2017.
 [21] G.E. Hinton, A. Krizhevsky, and S.D. Wang, “Transforming Autoencoders,” International Conference on Artificial Neural Networks, pp. 4451. Springer, Berlin, Heidelberg, 2011.

[22]
Y. LeCun,
“The MNIST database of Handwritten Digits,”
1998.