Improving the Robustness of Capsule Networks to Image Affine Transformations

Improving the Robustness of Capsule Networks to Image Affine Transformations

Jindong Gu
The University of Munich
Siemens AG, Corporate Technology
   Volker Tresp
The University of Munich
Siemens AG, Corporate Technology

Convolutional neural networks (CNNs) achieve translational invariance using pooling operations, which do not maintain the spatial relationship in the learned representations. Hence, they cannot extrapolate their understanding of the geometric transformation of inputs. Recently, Capsule Networks (CapsNets) have been proposed to tackle this problem. In CapsNets, each entity is represented by a vector and routed to high-level entities by a dynamic routing algorithm. The CapsNets have been shown to be more robust than CNNs to affine transformations of inputs. However, there is still a huge gap between their performance on transformed inputs compared to untransformed versions. In this work, we first revisit the routing procedure by (un)rolling its forward and backward passes. Our investigation reveals that the routing procedure contributes neither to generalization ability nor to the affine robustness of the CapsNets.

Furthermore, we explore the limitations of capsule transformations and propose affine CapsNets (Aff-CapsNets) that are more robust to affine transformations. On our benchmark task where models are trained on the MNIST dataset and tested on the AffNIST dataset, our Aff-CapsNets improve the benchmark performance by a large margin (from 79% to 93.21%), without using a routing mechanism. We also demonstrate the superiority of Aff-CapsNets on a real-world Brain Tumor Type classification task.

largesymbols”0D largesymbols”0D

1 Introduction

The human vision compensates for affine transformations, e.g., entities in an image and a rotated version of that image can both be recognized by the humen visual system, as long as the rotation is not too large. As popular computer vision models to understand images, Convolutional Neural Networks (CNNs) achieve affine robustness by training on a large amount of data that contain different transformations of target objects. Given limited training data, which is common in many real-world tasks, the robustness of CNNs to novel affine transformations is limited [22].

To learn features that are more aligned with human perception, Capsule Networks (CapsNets) have recently been proposed [22]. The proposed CapsNets differ from CNNs mainly in two special points: first, they represent each entity with a vector, the magnitude of which represents the probability of its existence in the image; second, they assign low-level entities to high-level ones using an iterative routing mechanism (a dynamic routing procedure). CapsNets aim to keep both equivariance of output pose vectors and invariance of output activations. The disentanglement of variation factors makes CapsNets more robust than CNNs to affine transformations.

The current benchmark task to evaluate the affine robustness of a model is to train the model on the standard MNIST dataset and test it on the AffNIST111Each example is an MNIST digit with a small affine transformation. dataset. CapsNets achieve 79% accuracy on AffNIST, while CNNs with similar network size only achieve 66% [22]. Although CapsNets show their superiority on this task, there is still a huge performance gap since CapsNets achieve more than 99% on the untransformed MNIST test dataset.

In our paper, we first investigate the effectiveness of components that make CapsNets robust to input affine transformations, with a focus on the routing algorithm. Many heuristic routing algorithms have been proposed [12, 24, 16] since [22] was published. However, recent work [18] shows that the proposed routing algorithms perform even worse than a uniform/random routing procedure.

From both numerical analysis and empirical experiments, our investigation reveals that the dynamic routing procedure contributes neither to the generalization ability nor to the affine robustness of CapsNets. We conclude that it is infeasible to improve the affine robustness by modifying the routing procedure. Instead, we investigate the limitations of the CapsNets architectures. In particular, we propose to apply identical transformation function for all primary capsules. We experimentally show that this improves the affine robustness of CapsNets significantly.

Our contributions can be summarized as follows: 1) We revisit the dynamic routing procedure of CapsNets; 2) We investigate the limitations of the current CapsNets architecture and propose a more robust affine Capsule Networks (Aff-CapsNet); 3) Based on extensive experiments, we investigate the properties of CapsNets trained without routing. Besides, we demonstrate the superiority of Aff-CapsNet.

The rest of this paper is organized as follows: Section 2 first reviews CapsNets and the related work. Section 3 investigates the effectiveness of the routing procedure by (un)rolling the forward and backward passes of the iterative routing iterations. Section 4 shows the limitation of current CapsNets on the affine transformations and proposes a robust affine CapsNet (Aff-CapsNet). Section 4 conducts extensive experiments to verify our findings and proposed modifications. The last two sections discuss and conclude our work.

2 Background and Related Work

In this section, we first introduce basic details about CapsNets with dynamic routing. Then, we review the related work.

2.1 Fundamentals of Capsule Networks

CapsNets [22] encode entities with capsules, each of which is represented by an activity vector (e.g., the activation of a group of neurons). The elements of each vector encode the properties of the corresponding entity. The length of the vector indicates the confidence of the entity’s existence. The output classes are represented as high-level capsules.

A CapsNet first maps the raw input features to low-level capsules and then routs the low-level capsules to high-level capsules. For instance, in image classification tasks, a CapsNet starts with one(or more) convolutional layer(s) that convert the pixel intensities into low-level visual entities. A following capsule layer of the CapsNet routs low-level visual entities to high-level visual entities. The CapsNet can have one or more capsule layers with routing procedures.

Given a low-level capsule of the -th layer with capsules, a high-level capsule of the -th layer with capsules, and a transformation matrix , the routing process is


where are a coupling coefficient that models the connection strength between and . The capsule is shrunk to a length in (0, 1) with a non-linear squashing function , which is defined as


The coupling coefficients are computed by an iterative routing procedure. They are updated so that the high agreement () corresponds to a high value of . The coupling coefficients between a -th capsule of the -th layer and all capsules of the -th layer sum to 1.


where initial logits are the log prior probabilities. They are replaced by in each iteration. The steps in Equations 1, 2, and 3 are repeated (specified manually) times in the routing process.

2.2 Related Work

Routing Algorithms Many papers have improved the routing-by-agreement algorithm. [25] generalizes existing routing methods within the framework of weighted kernel density estimation and proposes two fast routing methods with different optimization strategies. [8] proposes an attention-based routing procedure with an attention module, which only requires a fast forward-pass. The agreement between the capsules predicted by low-level capsules can also be calculated by Gaussian probability [12, 4] or distance measures [16] instead of the simple inner product.

Since the routing procedure is computationally expensive, several works propose solutions to avoid the complexity of the iterative routing process. [24] formulates the routing strategy as an optimization problem that minimizes a combination of clustering-like loss and a KL distance between the current coupling distribution and its last states. [17] approximates the expensive routing process with two branches: a master branch that collects primary information from its direct contact in the lower layer and an aide branch that replenishes the master branch based on pattern variants encoded in other lower capsules.

Understanding Routing Procedure [6] incorporates the routing procedure into the training process by making coupling coefficients trainable, which are supposed to be determined by an iterative routing process. The coupling coefficients are independent of examples, which stay unchanged in the testing phase. What they proposed is simply to reduce the iterative updates to a single forward pass with prior coupling coefficients. [2] removes the routing procedure completely and modifies the CapsNet architectures. Their pure CapsNets achieve competitive performance. However, it has not been investigated how the properties of their CapsNets will be affected by the removal of the routing procedure, e.g., the robustness to affine transformation. Furthermore, [18] shows that many routing procedures [22, 12, 24, 16] are heuristic, and perform even worse than a random routing assignment. Overall, the role of the routing procedure in CapsNets needs to be revisited systematically.

3 Revisiting Dynamic Routing of CapsNets

In this section, we analyze the dynamic routing, both theoretically and empirically. By unrolling the backpropagation of the routing procedure and rolling the forward propagation of the routing procedure, we show which role the routing procedure plays in the CapsNets.

Figure 1: The gradients of the loss w.r.t. randomly choosen are visualized. The blue lines correspond to the unrolled routing iterations in Gradient Backpropagation, while the yellow lines to rolled routing iterations.

3.1 Backpropagation through Routing Iterations

The forward pass of an iterative routing process can be written as the following iterative formulations


where the superscript is the index of an iteration, which is an integer starting from 1. The and are initialized as in Equation 3.

Given that there are K iterations and the classification loss of is , where is the prediction and the target, the gradients through the routing procedure are


The gradients are propagated through the unrolled routing iteration via the second item of Equation 5, which is also the main computational burden of the expensive routing procedure in CapsNets. By unrolling this term, we prove that


where C is a constant, which can be integrated into the learning rate in the optimization process (see the proof in Appendix A). The approximation means the gradients flowing through in Equation 5 can be ignored. The can be treated as a constant in Gradient Backpropagation, and the routing procedure can be detached from the computational graph of CapsNets.

To confirm Equation 6 empirically, we visualize . Following [22], we train a CapsNet on the MNIST dataset. The architecture and the hyper-parameter values can be found in Appendix B. We first select capsule predictions randomly before the routing process and then visualize their received gradients in two cases: 1) unrolling the routing iterations as in [22]; 2) rolling the routing iterations by taking as a constant in Gradient Backpropagation (i.e., ignoring the second item in Equation 5). As shown in each plot of Figure 1, the gradients of the two cases (blue lines and yellow lines) are similar to each other.

In this section, we aim to show that the intrinsic contribution of the routing procedure is to identify specified constants as coupling coefficients . Without a doubt, both computational cost and memory footprint can be saved by rolling the routing iterations in Gradient Backpropagation. The computational graphs of the two cases can be found in Appendix C.

3.2 Forward Pass through Routing Iterations

The forward iterative routing procedure can be formulated as a function, mapping capsule predictions to coupling coefficients, i.e., where the indexes of low-level capsules vary from 1 to N and the indexes of high-level capsules vary from 1 to M. Given an instance, without loss of generality, we assume the ground-truth class is M (i.e., ). The optimal coupling coefficients of the instance are


where the first term describes the agreement on the target class, and the second term corresponds to the agreement on non-ground-truth classes. The optimal coupling coefficient corresponds to the case where the agreement on the target class is maximized, and the agreement on the non-ground-truth classes is minimized.

Figure 2: The green lines correspond to the model with dynamic routing, while the magenta ones to the model without routing procedure. For both models, the agreement on the target class increases with training time, and the agreement on the non-ground-truth classes decreases. The values are averaged over the whole training or test dataset.

Many routing algorithms differ only in how they approximate . For instance, the original work [22] approximates with an iterative routing procedure. Without requiring iterative routing steps, [6] makes trainable to approximate . Their proposal can be understood as only one-step routing with learned prior coupling coefficients. By further reformulation, we show that the optimal can be learned without coupling coefficients.


The connection strengths between low-level capsules and high-level capsules can be learned implicitly in the transformation matrix . Therefore, we can conclude that different ways to approximate do not make a significant difference since the coupling coefficients will be learned in the transformation matrix implicitly.

We visualize the implicit learning process of the coupling coefficients. In our experiments, we remove the iterative routing procedure by setting all coupling coefficient as a constant . In each training epoch, the agreement on the target class and on the non-ground-truth classes is visualized in Figure 2, respectively. As a comparison, we also visualize the corresponding agreement values of CapNets with the dynamic routing process. We can observe that, during training process, the agreement on the target class increases (in the left plot) for both cases, and the agreement on the non-ground-truth classes decreases (in the right plot). In other words, increases in both CapNets with/without routing procedure, meaning that the coupling coefficients can be learned implicitly.

In summary, the affine robustness of CapsNet can not be contributed to the routing procedure. We conclude that it is not infeasible to improve the robustness of CapsNet by modifying the current routing-by-agreement algorithm.

Figure 3: The overview of Capsule Network. The transformation matrix only can transform to high-level capsules, while does not. When an input is transformed (e.g., rotated), the receptive field corresponding to is not any more. For the new unexpected , the transformation using can fail.

4 Affine Robustness of Capsule Networks

Besides the dynamic routing process, the other difference between CapsNets and traditional CNNs is the CapsNets architecture. CapsNets represent each entity with a capsule and transform it to high-level entity with transformation matrices. In this section, we investigate the limitation of the transformation process in terms of affine robustness and propose robust affine capsule networks.

4.1 The Limitation of CapsNets

The architecture overview of a CapsNet [22] is shown in Figure 3. The CapsNet starts with two convolutional layers, which converts the pixel intensities to form primary (low-level) capsules (e.g., the blue cuboids in the figure is a capsule ). Each primary capsule has a certain receptive field (e.g., the image patch marked with the yellow rectangle). For all inputs, the coordinates of the receptive field of are the same. In other words, a primary capsule can only see a specific area in input images. The corresponding converting process is .

Each primary capsule is transformed to high-level capsules with the corresponding transformation matrix (e.g., ). Each transformation matrix learns how to transform the corresponding low-level capsule to high-level ones, e.g., the transformation process corresponds to the capsule is . Formally, the votes from the primary capsule are


The transformation matrix can only make meaningful transformations for the entities that have ever appeared in the position of . The input domain of the transformation function is .

In the testing phase, if novel affine transformations are conducted on the input, the votes are not meaningful since is not in the domain . In other words, the transformation matrix does not make meaningful transformation since the patch that has never appeared in the position of the patch during training. Hence, the performance of CapsNet is limited to novel affine transformations of inputs.

4.2 Robust Affine Capsule Networks

To overcome the limitation above, we propose a simple and efficient solution. Concretely, we propose to use the same transformation function for all primary capsules (i.e., ensuring ). We implement a robust affine capsule network (Aff-CapsNet) by sharing a transformation matrix. Formally, for Aff-CapsNets, we have


where N is the number of primary capsules. In Aff-CapsNets, the transformation matrix can make a meaningful transformation for all primary capsules since it learns how to transform all low-level capsules to high-level capsules during training. The transformation matrix sharing has also been applied in an existing publication [20]. The difference is that they aim to save parameters, while our goal is to make CapsNets more robust to affine transformations.

From another perspective, primary capsules and high-level capsules correspond to local coordinate systems and global ones, respectively. A transformation matrix is supposed to map a local coordinate system to the global one. One might be wondering that the transformation from each local coordinate system to a global one requires a specific transformation matrix. In existing architectures, the coordinate system is high-dimensional. Hence, a single shared transformation matrix is able to make successful transformations for all local coordinate systems.

5 Experiments and Analysis

The experiments include three parts: 1) We train CapsNets with different routing mechanisms (including no routing) on popular standard datasets and compare their properties from many perspectives; 2) We show that Aff-CapsNets outperform CapsNets on the benchmark dataset and achieves state-of-the-art performance; 3) We demonstrate the superiority of Aff-CapsNets on a real-world Brain Tumor Type classification task. For all the experiments of this section, we train models with 5 random seeds and report their averages and variances.

Dynamic-R 99.41( 0.08) 92.12( 0.29) 91.32( 0.19) 74.64( 1.02)
Rolled-R 99.29( 0.09) 91.53( 0.22) 90.75( 0.52) 74.26( 0.94)
Trainable-R 99.55( 0.04) 92.58( 0.10) 92.37( 0.29) 76.43( 1.11)
No-R 99.54( 0.04) 92.53( 0.26) 92.15( 0.29) 76.28( 0.39)
Table 1: The performance of CapsNets with different routing procedures on standard datasets.

5.1 Effectiveness of the Dynamic Routing

In Section 3, we show that the routing mechanism can be learned implicitly in CapsNets without routing procedure. Our experiments in this section aim to show that the advantages of CapsNets does disappear when trained with no routing. The following training routines are considered:

  1. Dynamic-R) with standard dynamic routing in [22];

  2. Rolled-R) with a rolled routing procedure by treating coupling coefficients as constants during Gradient Backpropagation, as analyzed in Section 3.1;

  3. Trainable-R) one-step routing with trainable coupling coefficients, as in [6];

  4. No-R) without routing procedure, which is equivalent to the uniform routing in [18, 2].

Figure 4: Disentangled Individual Dimensions of Capsules: By perturbing one dimension of an activity vector, the variations of an input image are reconstructed.
Figure 5: The average reconstruction loss of CapsNets with Dynamic-R and No-R on the test dataset is shown in each epoch of the training process.

We train CapsNets on the standard datasets with different routing procedures described above. The performance is reported in Table 1. Given the variance corresponding to each model, the difference between them is relatively small. The reason behind this is that coupling coefficients can be learned in transformation matrices implicitly, and all the models possess an transformation process. The models trained with No-R do not prevent the learning of coupling coefficients. We can also observe that the models with Trainable-R or No-R show a slightly better performance than the other two. To our understanding, the reason is that they do not suffer the polarization problem of coupling coefficients [17].

From this experiment, we can only conclude that the routing procedure does not contribute to the generalization ability of CapsNets. The CapsNets show many superior properties over CNNs, not only the classification performance. In the following, we analyze the properties of CapsNets with No-R and compare them with CapsNets with Dynamic-R.

5.1.1 On learned Representations of Capsules

When training CapsNets, the original input is reconstructed from the activity vector (i.e., instantiation parameters) of a high-level capsule. The reconstruction is treated as a regularization technique. In CapsNets with Dynamic-R [22], the dimensions of the activity vector learn how to span the space containing large variations. To check such property of CapsNets with No-R, following [22], we feed a perturbed activity vector of the ground-truth class to decoder network.

The perturbation of the dimensions can also cause variations of the reconstructed input. We show some examples in Figure 5. The variations include stroke thickness, width, translation, rotation, and various combinations. In Figure 5, we also visualize the reconstruction loss of the models with Dynamic-R and the ones with No-R. The CapsNets with No-R show even less reconstruction error, and can reconstruct inputs better.

5.1.2 Parallel Attention Mechanism between Capsules

Dynamic routing can be viewed as a parallel attention mechanism, in which each high-level capsule attends to some active low-level capsules and ignores others. The parallel attention mechanism allows the model to recognize multiple objects in the image even if objects overlap [22]. The superiority of the parallel attention mechanism can be shown on the classification task on MultiMNIST dataset [11, 22]. Each image in this dataset contains two highly overlapping digits. CapsNets with dynamic routing procedure shows high performance on this task.

In this experiment, we show that the parallel attention mechanism between capsules can be learned implicitly without the routing mechanism. Following the experimental setting in [22], we train a CapsNet with No-R on the same classification task of classifying highly overlapping digits. The model No-R achieves 95,49% accuracy on the test set, while the one with Dynamic-R achieves 95% accuracy. The removal of the routing procedure does not make the parallel attention mechanism of CapsNets disappear.

Models Test on MNIST Test on AffNIST
CNN [22] 99.22% 66%
Dynamic-R [22] 99.23% 79%
No-R 99.22% 81.81%
Table 2: The performance on the expanded MNIST test set and the AffNIST test set.

5.1.3 Robustness to Affine Transformation

CapsNets are also known for their robustness to affine transformation. It is important to check whether the removal of the routing procedure affects the affine robustness. We conduct experiments on a standard benchmark task. Following [22], we train CapsNets with or without routing procedure on MNIST training dataset and test them on affNIST dataset, respectively. The images in MNIST training dataset are placed randomly on a blackground of pixels to match the size of images in affNIST dataset. The CNN baseline is set the same as in [22].

(a) Without a routing procedure: the test accuracy of CapsNets and Aff-Capsnets on on the expanded MNIST test set and the AffNIST test set.
(b) With the dynamic routing: the test accuracy of CapsNets and Aff-Capsnets on the expanded MNIST test set and the AffNIST test set.
Figure 6: For both cases (with or without routing procedure), Aff-CapsNets clearly outperform CapsNets on the AffNIST dataset.

It is hard to decide if one model is better at generalizing to novel affine transformations than another when they achieved different accuracy on untransformed examples. To eliminate this confounding factor, we stopped training the models when they achieve similar performance, following [22]. The performance is shown in Table 2. Without routing procedure, the CapsNets show even better affine robustness.

In summary, our experiments show that the dynamic routing procedure contributes neither to the generalization ability nor to the affine robustness. Since the high affine robustness of CapsNet cannot be attributed to the routing procedure, it is the inductive bias (architecture) of CapsNets that contributes to the affine robustness.

5.2 Affine Robustness of Aff-CapsNets

In Section 4, we propose Aff-CapsNets that are more robust to the novel affine transformations of inputs. In this experiment, we train Aff-CapsNets with Dynamic-R and No-R respectively. As a comparison, we also train CapsNets with or without dynamic routing correspondingly.

We visualize the test accuracy on the expanded MNIST test set and the AffNIST test set. The performance is shown in Figure 6. The lines show the averaged values, while the colored areas around lines describe the variances caused by different seeds. Figure 5(a) shows the accuracy of models trained without a routing procedure. We can observe that the Aff-CapsNets constantly shows better accuracy than CapsNets on AffNIST. To a great extent, our Aff-CapsNets covers the performance gap between the test accuracy on untransformed examples and that on transformed ones.

In addition, the Aff-CapsNets architecture is still effective, even when the dynamic routing is applied in training (see Figure 5(b)). We can also observe that the CapsNets with dynamic routing overfit to the current viewpoints. With the training process going on, the coupling coefficients are polarized (become close to 0 or 1) [17]. The polarization of the coupling coefficient causes the overfitting. Furthermore, the training with dynamic routing is more unstable than without routing. The variance of model test performance in Figure 5(b) is much bigger than the ones in Figure 5(a).

Models Trained on AffNIST? MNIST AffNIST
Marginal. CNN [26] Yes 97.82% 86.79%
TransRA CNN[3] Yes 99.25 % 87.57%
BCN [5] Mix* 97.5% 91.60%
CNN [22] No 99.22% 66%
Dynamic-R [22] No 99.23% 79%
GE-CAPS [16] No - 89.10%
SPARSECAPS [21] No 99% 90.12%
Aff-CapsNet + No-R No 99.23% 93.21(0.65)%
Table 3: Comparison to state-of-the-art performance on the benchmark task.

We now compare our model with previous work. In Table 3, we list the performance of CNN variants of and CapsNet variants on this task. Without training on AffNIST dataset, our Aff-CapsNets achieve state-of-the-art performance on AffNIST test dataset. This experiment shows that the proposed model is robust to input affine transformation.

Figure 7: Images from the Brain Tumor Type classification test dataset: Rotation, Transformation, and Scaling of images can be observed.
Models #Parameters Test Accuracy
CNN [1] 9.4M 61,97%
CapsNet + Dynamic-R 8.23M 81.36( 0.26)%
CapsNet + No-R 8.23M 81.50( 0.34)%
Aff-CapsNet + Dynamic-R 6.75M 83.57( 0.25)%
Aff-CapsNet +No-R 6.75M 83.94( 0.46)%
Table 4: The number of parameters of each model and their performance on the Brain Tumor Type classification test set.

5.3 Brain Tumor Type Classification

In this experiment, we show that Aff-CapsNets outperform CapsNets on a real-world classification task. The dataset used in the experiment is a Brain Tumor Type Classification dataset [7], which contains 3064 T1-weighted contrast-inhanced images with three kinds of brain tumor from 233 patients. Following the experimental setup in [7, 1], the 233 patients are randomly split into 5 subsets of roughly equal size. It is notable that slices from the same patient will not appear in different subsets simultaneously. For all the experiments, five-fold cross-validation is used. In the cross-validation, one subset is sequentially used as the test set, whereas the remaining four subsets are used as the training set. The average of five validation accuracy is used as a result. The scores of five runs are averaged and reported as the final score.

The MRI images in the dataset are visualized in Figure 7. Different affine transformations can be observed. The images are down-sampled to as model inputs. We train CapsNets and Aff-CapsNets on the data, respectively. The hyperparameters are set the same as in [22]. The result is reported in Table 4. With fewer parameters, Aff-CapsNet outperforms CapsNet by 2,21% (with dynamic routing) or 2,44% (without routing). The classification performance on the cross-validation set is limited not only by affine transformations but also by the generalization ability of the models to new patients. Hence, our improvement on this task is only limited. However, this experiment clearly shows the superiority of Aff-CapsNets.

6 Discussion

Going Deeper with CapsNets One way to make CapsNets deep is to integrate advanced techniques of training CNNs into CapsNets. Some integrations have been proven to be successful, e.g., integration of skip connections [10, 20] and dense connections [13, 19]. Instead of blindly integrating more advanced techniques from CNN into CapsNets, it is necessary to investigate more into the effective components in CapsNets. Our investigation reals that the dynamic routing procedure contributes neither to the generalization ability nor to the affine robustness of CapsNets. Such conclusion is helpful to train CapsNets on large scale dataset, e.g., Imagenet 1k dataset [9].

Application of CapsNets to Computer Vision Tasks Besides the object recognition task, CapsNets are also applied to many other computer vision tasks, for examples, object segmentation [15], image generative models [14, 23], and adversarial defense [12]. It is not clear whether routing procedures are necessary for their tasks. If not, the architectures of CapsuleNets can be integrated into these vision tasks with much less effort.

The Necessity of the Routing Procedure in CapsNets Although [22] shows many advantages of CapsNets with dynamic routing over CNNs, our investigation shows that all the advantages do not disappear when the routing procedure is removed. However, we would not propose to conclude that the routing procedure is totally unnecessary in CapsNets. Instead, we raise an important question to the community. What the routing procedure is really good for? If the routing procedure is not necessary for a given task, we could save a large amount of time and the computational cost required by the routing procedures.

7 Conclusion

We revisit the dynamic routing procedure of CapsNets. Our numerical analysis and extensive experiments show that neither the generalization ability nor the affine robustness of CapsNets is reduced by removing the dynamic routing procedure. The conclusion guides us to focus on CapsNets architecture instead of various routing procedures to improve the affine robustness. By exploring the limitation of existing architecture, we propose Aff-CapsNets, which improves affine robustness with much fewer parameters.

Since this work mainly focuses on the robustness to affine transformation, we only investigate the superior properties of Capsule with dynamic routings. In other variants of CapsNets, other advantages are also shown, e.g., the adversarial robustness, and viewpoint invariance. We will address these issues in future work.


  • [1] P. Afshar, A. Mohammadi, and K. N. Plataniotis (2018) Brain tumor type classification via capsule networks. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3129–3133. Cited by: §5.3, Table 4.
  • [2] Anonymous (2020) Capsule networks without routing procedures. In Submitted to International Conference on Learning Representations, Note: under review External Links: Link Cited by: §2.2, item 4.
  • [3] S. Asano (2018) Proposal of transformation robust attentive convolutional neural network. Ph.D. Thesis, Waseda University. Cited by: Table 3.
  • [4] M. T. Bahadori (2018) Spectral capsule networks. In ICML Workshop, Cited by: §2.2.
  • [5] S. Chang, J. Yang, S. Park, and N. Kwak (2018) Broadcasting convolutional network for visual relational reasoning. In ECCV, pp. 754–769. Cited by: Table 3.
  • [6] Z. Chen and D. Crandall (2019) Generalized capsule networks with trainable routing procedure. In ICML Workshop, Cited by: §2.2, §3.2, item 3.
  • [7] J. Cheng, W. Huang, S. Cao, R. Yang, W. Yang, Z. Yun, Z. Wang, and Q. Feng (2015) Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS one 10 (10), pp. e0140381. Cited by: §5.3.
  • [8] J. Choi, H. Seo, S. Im, and M. Kang (2019) Attention routing between capsules. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0. Cited by: §2.2.
  • [9] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In CVPR, pp. 248–255. Cited by: §6.
  • [10] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In CVPR, pp. 770–778. Cited by: §6.
  • [11] G. E. Hinton, Z. Ghahramani, and Y. W. Teh (2000) Learning to parse images. In Advances in neural information processing systems, pp. 463–469. Cited by: §5.1.2.
  • [12] G. E. Hinton, S. Sabour, and N. Frosst (2018) Matrix capsules with em routing. In ICLR, Cited by: §1, §2.2, §2.2, §6.
  • [13] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger (2017) Densely connected convolutional networks. In CVPR, pp. 4700–4708. Cited by: §6.
  • [14] A. Jaiswal, W. AbdAlmageed, Y. Wu, and P. Natarajan (2018) Capsulegan: generative adversarial capsule network. In ECCV, pp. 0–0. Cited by: §6.
  • [15] R. LaLonde and U. Bagci (2018) Capsules for object segmentation. In International Conference on Medical Imaging with Deep Learning, Cited by: §6.
  • [16] J. E. Lenssen, M. Fey, and P. Libuschewski (2018) Group equivariant capsule networks. In Advances in Neural Information Processing Systems, pp. 8844–8853. Cited by: §1, §2.2, §2.2, Table 3.
  • [17] H. Li, X. Guo, B. DaiWanli Ouyang, and X. Wang (2018) Neural network encapsulation. In ECCV, pp. 252–267. Cited by: §2.2, §5.1, §5.2.
  • [18] I. Paik, T. Kwak, and I. Kim (2019) Capsule networks need an improved routing algorithm. ArXiv abs/1907.13327. Cited by: §1, §2.2, item 4.
  • [19] S. S. R. Phaye, A. Sikka, A. Dhall, and D. R. Bathula (2018) Multi-level dense capsule networks. In Asian Conference on Computer Vision, pp. 577–592. Cited by: §6.
  • [20] J. Rajasegaran, V. Jayasundara, S. Jayasekara, H. Jayasekara, S. Seneviratne, and R. Rodrigo (2019) DeepCaps: going deeper with capsule networks. In CVPR, pp. 10725–10733. Cited by: §4.2, §6.
  • [21] D. Rawlinson, A. Ahmed, and G. Kowadlo (2018) Sparse unsupervised capsules generalize better. arXiv preprint arXiv:1804.06094. Cited by: Table 3.
  • [22] S. Sabour, N. Frosst, and G. E. Hinton (2017) Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856–3866. Cited by: §1, §1, §1, §1, §2.1, §2.2, §3.1, §3.2, §4.1, item 1, §5.1.1, §5.1.2, §5.1.2, §5.1.3, §5.1.3, §5.3, Table 2, Table 3, §6.
  • [23] R. Saqur and S. Vivona (2019) Capsgan: using dynamic routing for generative adversarial networks. In Science and Information Conference, pp. 511–525. Cited by: §6.
  • [24] D. Wang and Q. Liu (2018) An optimization view on dynamic routing between capsules. In ICLR Worksop, Cited by: §1, §2.2, §2.2.
  • [25] S. Zhang, Q. Zhou, and X. Wu (2018) Fast dynamic routing based on weighted kernel density estimation. In International Symposium on Artificial Intelligence and Robotics, pp. 301–309. Cited by: §2.2.
  • [26] J. Zhao, J. Li, F. Zhao, X. Nie, Y. Chen, S. Yan, and J. Feng (2017) Marginalized cnn: learning deep invariant representations. In BMVC, Cited by: Table 3.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description