Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition

Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition

Heliang Zheng , Jianlong Fu, Zheng-Jun Zha, Jiebo Luo
University of Science and Technology of China, Hefei, China
Microsoft Research, Beijing, China
University of Rochester, Rochester, NY
zhenghl@mail.ustc.edu.cn, zhazj@ustc.edu.cn, jianf@microsoft.com, jluo@cs.rochester.edu
This work was performed when Heliang Zheng was visiting Microsoft Research as a research intern.Corresponding author.

Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps by modeling the inter-channel relationships, 2) an attention-based sampler which highlights attended parts with high resolution, and 3) a feature distiller, which distills part features into an object-level feature by weight sharing and feature preserving strategies. Extensive experiments verify that TASN yields the best performance under the same settings with the most competitive approaches, in iNaturalist-2017, CUB-Bird, and Stanford-Cars datasets.


1 Introduction

Fine-grained visual categorization (FGVC) focuses on distinguishing subtle visual differences within a basic-level category (e.g., bird [1, 34] and car [13, 20, 36]). Although the techniques of convolutional neural network (CNN) [8, 15, 25] for general image recognition [14, 23] have become increasingly practical, FGVC is still a challenging task where discriminative details are too subtle to be well-represented by traditional CNN. Thus the majority of efforts in the fine-grained community focuses on learning better representation for such subtle yet discriminative details.

Figure 1: An illustration of learning discriminative details by TASN for a “bule jay.” As shown in (b), TASN learns such subtle details by up-sampling each detail into high-resolution. And the white concentric circles in (c) indicates fine-grained details.

Existing attention/part-based methods [2, 7, 33, 40] try to solve this problem by learning part detectors, cropping and amplifying the attended parts, and concatenating part features for recognition. Although promising performance has been achieved, there are several critical issues in such a pipeline. Specifically, 1) the number of attention is limited and pre-defined, which restricts the effectiveness and flexibility of the model. 2) Without part annotations, it is difficult to learn multiple consistent (i.e., attending on the same part for each sample) attention maps. Although a well-designed initialization [7, 16, 40] can benefit the model training, it is not robust and cannot handle the cases with uncommon poses. Moreover, 3) training CNNs for each part is not efficient. Such problems evolve as bottlenecks for the study on attention-based methods.

To address the above challenges, we propose a trilinear attention sampling network (TASN) which learns fine-grained details from hundreds of part proposals and efficiently distills the learned features into a single convolutional neural network. The proposed TASN consists of a trilinear attention module, an attention-based sampler, and a feature distiller. First, the trilinear attention module takes as input feature maps and generates attention maps by self-trilinear product, which integrates feature channels with their relationship matrix. Since each channel of feature maps is transformed into an attention map, hundreds of part proposals can be extracted. Second, attention-based sampler takes as input an attention map as well as an image, and highlights attended parts with high resolution. Specifically, for each iteration, the attention-based sampler generates a detail-preserved image based on a randomly selected attention map, and a structure-preserved image based on an averaged attention map. The former learns fine-grained feature for a specific part, and the latter captures global structure and contains all the important details. Finally, A part-net and a master-net are further formulated as “teacher” and “student,” respectively. Part-net learns fine-grained features from the detail-preserved image and distills the learned features into master-net. And the master-net takes as input the structure-preserved image and refines a specific part (guided by the part-net) in each iteration. Such distillation is achieved by weight sharing and feature preserving strategies. Note that we adopt knowledge distilling introduced in [10] instead of concatenating part features, because the part number is large and not pre-defined.

Since the feature distiller transfers the knowledge from part-net into master-net via optimizing the parameters, 1) stochastic details optimization (i.e., randomly optimize one part in each iteration) can be achieved, which makes it practical to learn details from hundreds of part proposals, and 2) efficient inference can be obtained as we can use master-net to perform recognition in the testing stage. To the best of our knowledge, this work makes the first attempt to learn fine-grained features from hundreds of part proposals and represent such part features with a single convolutional neural network. Our contributions are summarized as follows:

  • We propose a novel trilinear attention sampling network (TASN) to learn subtle feature representations from hundreds of part proposals for fine-grained image recognition.

  • We propose to optimize TASN in a teacher-student manner, in which fine-grained features can be distilled into a single master-net with high-efficiency.

  • We conduct extensive experiments on three challenging datasets (iNaturalist, CUB Birds and Stanford Cars), and demonstrate that TASN outperforms part-ensemble models even with a single stream.

The remainder of the paper is organized as follows. We describe related work in Section 2, and introduce our proposed TASN model in Section 3. An evaluation on three widely-used datasets is presented in Section 4, followed by conclusions in Section 5.

Figure 2: Overview of the proposed Trilinear Attention Sampling Network (TASN). The trilinear attention module in (b) takes as input convolutional feature maps (denoted as “conv”), and generates attention maps (denoted as “att”). The attention sampling module in (c) further takes as input an attention map as well as the original image to obtain sampled images. Specifically, average pooling and random selection (in each iteration) are conducted over attention maps to obtain structure preserved image in (d) and detail preserved image in (e), respectively. The part-net (in green) learns fine-grained features from (e) and generates a soft target to distill such features into the master-net (in blue) via soft target cross entropy [10]. [Best viewed in color]

2 Related Works

Attention Mechanism: As subtle yet discriminative details play an important role for Fine-Grained Image Recognition, learning to attend on discriminative parts is the most popular and promising direction. Thus various of attention mechanisms have been proposed in recent years [7, 19, 26, 35, 40]. DT-RAM [19] proposed a dynamic computational time model for recurrent visual attention, which can attend on the most discriminative part in dynamic steps. RA-CNN [7] proposed a recurrent attention convolutional neural network to recurrently learn attention maps in multiple (i.e., 3) scales. And MA-CNN [40] takes one step further to generate multiple (i.e., 4) consistency attention maps in a single scale by designing a channel grouping module. However, the attention numbers (i.e., 1, 3, 4, respectively) are pre-defined, which counts against the effectiveness and flexibility of the model.

Meanwhile, high-order attention methods are proposed in visual question answering (VQA) and video classification. Specifically, BAN [12] proposed a bilinear attention module to handle the relationship between image regions and the words in question, and Non-local [30] calculates the dot production of features to represent the spatial and temporary relationship in video frames. Different from these works, our trilinear attention module conducts bilinear pooling to obtain the relationship among feature channels, which is further utilized to integrate such features to obtain third-order attention maps.

Adaptive Image Sampling: To preserve fine-grained details for recognition, high input resolution ( v.s. ) is widely adopted [5, 33, 40] and it can significantly improve the performance [5]. However, high resolution brings large computational cost. More importantly, the importance of different regions are various, while directly zooming in images cannot promise different regions with different resolutions. STN [11] proposed a non-uniformed sampling mechanism which performs well on MNIST datasets [17]. But without explicit guidance, it is hard to learn non-uniformed sampling parameters for sophisticated tasks such as fine-grained recognition, thus they finally learned two parts without non-uniformed sampling. SSN [22] firstly proposed to use saliency maps as the guidance of non-uniformed sampling and obtained significant improvements. Different from them, our attention sampler 1) conduct non-uniformed sampling based on trilinear attention maps, and 2) decomposes attention maps into two dimensions to reduce spatial distortion effects.

Knowledge Distilling: Knowledge distilling is firstly proposed by Hinton et al. [10] to transfer knowledge from an ensemble or from a large highly regularized model into a smaller, distilled model. The main idea is using soft targets (i.e., the predicted distribution of ensemble/large model) to optimize the small model, for it contains more information than the one-hot label. Such a simple yet effective idea inspires many researchers and has been further studied by [9, 38]. In this paper, we adopt this technique to distill the learned details from part-net into master-net.

3 Method

In this section, we introduce the proposed Trilinear Attention Sampling Network (TASN), which is able to represent rich fine-grained features by a single convolutional neural network. TASN contains three modules, i.e., a trilinear attention module for details localization, an attention-based sampler for details extraction, and a feature distiller for details optimization.

An overview of the proposed TASN is shown in Figure 2. Given an input image in (a), we first take it through several convolutional layers to extract feature maps, which is further transformed into attention maps by the trilinear attention module in (b). To learn fine-grained features for a specific part, we randomly select an attention map and conduct attention sampling over the input image using the selected attention map. The sampled image in (e) is named as detail-preserved image since it can preserve a specific detail with high resolution. Moreover, to capture global structure and contain all the important details, we average all the attention maps and again conduct attention sampling, such a sampled image in (d) is called structure-preserved image. We further formulate a part-net to learn fine-grained representation for detail-preserved images, and a master-net to learn the features for the structure-preserved image. Finally, the part-net generates soft targets to distill the fine-grained features into master-net via soft target cross entropy [10].

3.1 Details Localization by Trilinear Attention

In this subsection, we introduce our trilinear attention module, which transfers convolutional feature maps into attention maps. As shown in previous work [24, 39], each channel of the convolutional features corresponds to a visual pattern, however, such feature maps cannot act as attention maps due to the lack of consistency and robustness [32, 40]. Inspired by [40], we transform feature maps into attention maps by integrating feature channels according to their spatial relationship. Note that such a process can be implemented in a trilinear formulation, thus we call it trilinear attention module.

Given an input image , we extract convolutional features by feeding it into multiple convolutional, batch normalization, ReLU, and pooling layers. Specifically, we use resnet-18 [8] as backbone. To obtain high-resolution feature maps for precise localization, we remove the two down-sampling processes from original resnet-18 by changing convolutional stride. Moreover, to improve the robustness of convolutional response, we increase the field of views [3] by appending two sets of dilated convolutional layers with multiple dilate rates. In the training stage, we added a softmax classifier to optimize such convolutional features.

Assume that the feature maps is a tube with a dimension of , where , and indicate channel numbers, height, and width respectively. We reshape this feature into a matrix with a shape of , which is denoted as . Then our trilinear function can be basically formulated as:


where is the bilinear feature, which indicates the spatial relationship among channels. Specifically, is the channel of feature maps, and indicates the spatial relationship between channel and channel . To make feature maps more consistency and robust, we further integrate spatial relationship into feature maps by conducting dot production over and , thus trilinear attention maps can be obtained (which is shown in Figure 3).

We further studied different normalization methods to improve the effectiveness of trilinear attention, and a detailed discussion can be found in Section 4.2. To the end, we adopt the following normalized trilinear attention:


where indicates normalization over the second dimension of a matrix. Note that these two normalization functions have different meanings: The first one is spatial normalization which keeps each channel of feature maps within the same scale. And the second one is relationship normalization which is conducted over each relationship vector . We denote the output of the trilinear function in Equation 2 as , i.e., . Finally, we reshape into the shape of , thus each channel of indicates an attention map .

Figure 3: An illustration the trilinear product. indicates convolutional feature maps, and we can obtain inter-channel relationships by . After that, we integrate each feature map with its related ones to get trilinear attention maps via conducting dot production over and .

3.2 Details Extraction by Attention Sampling

In this subsection, we introduce our attention-based sampler, which takes as input an image as well as trilinear attention maps, and generates a structure-preserved image and a detail-preserved image. The structure-preserved image captures the global structure and contains all the important details. Compared to the original image, the structure-preserved one removed the regions without fine-grained details, thus the discriminative parts can be better represented with high resolution. The detail-preserved image focuses on a single part, which can preserve more fine-grained details.

Given an image , we obtain structure-preserved image and detail-preserved image by conducting non-uniform sampling over different attention maps:


where is the attention maps, indicates the non-uniform sampling function, indicates average pooling over channels, and indicates randomly selecting a channel from the input. We calculate the average of all attention maps to guide structure-preserved sampling, because such an attention map takes all the discriminative parts into consideration. And we randomly select one attention map for detail-preserved sampling, thus it can preserve the fine-grained details of this attended area with high resolution. With the training process going on, all attention maps have the opportunity to be selected, thus different fine-grained details can be asynchronously refined.

Our basic idea for attention-based sampling is considering the attention map as probability mass function, where the area with large attention value is more likely to be sampled. Inspired by the inverse-transform [6], we implement the sampling by calculating the inverse function of the distribution function. Moreover, we decompose attention maps into two dimensions to reduce spatial distortion effects.

Taking structure-preserved sampling for example, we first calculate the integral of the structure-preserved attention map over and axis:


where and are the width and height of the attention map, respectively. Note that we use function to decompose the attention map into two dimensions, because it is more robust than the alternative . We can further obtain the sampling function by:


where indicates the inverse function of . In a word, the attention map is used to calculate the mapping function between the coordinates of the original image and the sampled image.

Such a sampling mechanism is illustrated in Figure  4. Given an attention map in (a), we first decompose the map into two dimensions by calculating the max values over axis (b1) and axis (b2). Then the integral of (b1) and (b2) are obtained and shown in (c1) and (c2), respectively. We further calculate the inverse function of (c1) and (c2) in a digital manner, i.e., we uniformly sample points over the axis, and follow the red arrow (shown in (c1) and (c2)), and the blue arrow to obtain the values over axis. (d) shows the sampling points by blue dots, and we can observe that the regions with large attention values are allocated with more sampling points. Finally, (e) shows the result of the sampled image. Note that the example in Figure 4 is a structure-preserved sampling case.

Figure 4: An example of attention-based non-uniform sampling. (a) is an attention map with Gaussian distribution. (b1) and (b2) are the marginal distributions over and axis, respectively. (c1) and (c2) are the integrals of marginal distributions. (d) shows the sampling points by the blue dot, and (e) illustrates the sampled image. [Best viewed in color with zoom-in.]

3.3 Details Optimization by Knowledge Distilling

In this subsection, we introduce our details distiller, which takes as input a detail-preserved image and a structure-preserved image, and transfers the learned details from part-net to master-net in a teacher-student manner.

Specifically, for each iteration, the attention-based sampler introduced in Section  3.2 can provide a structure-preserved image (denoted as ) and a detail-preserved one (denoted as ). We first obtain the fully connected (fc) outputs by feeding these two images into the same backbone CNN (e.g., Resnet-50 [8]). The fc outputs are denoted as and , respectively. Then the “softmax” classifier converts and into a probability vector and , which indicates the predicted probability over each class. Taking for example:


where is a parameter namely temperature, which is normally set to 1 for classification tasks. While in knowledge distilling, a large value for is important as it can produce a soft probability distribution over classes. We obtain the soft target cross entropy [10] for the master-net as:


where denotes the class number. Finally, the objective function of the master-net can be drived by:


where represents the classification loss function, is a one hot vector which indicates the class label and denotes loss weight of the two terms. The soft target cross entropy aims to distill the learned feature for fine-grained details and transfer such information to the master-net. As the attention-based sampler randomly select one part in each iteration, all the fine-grained details can be distilled to the master-net in training process. Note that the convolutional parameters are shared for part-net and master-net, which is important for distilling, while the sharing of fully connected layers is optional.

4 Experiments

4.1 Experiment setup

Dataset # Class # Train # Test
CUB-200-2011 [34] 200 5,994 5,794
Stanford-Car [13] 196 8,144 8,041
iNaturalist-2017 [27] 5,089 579,184 95,986
Table 1: Detailed statistics of the three datasets used in this paper.

Datasets: To evaluate the effectiveness of our proposed TASN, we conducted experiments on three extensive and competitive datasets, namely Caltech-UCSD Birds (CUB-200-2011) [34], Stanford Cars [13] and iNaturalist-2017[27], respectively. The detailed statistics with category numbers and the standard training/testing splits can be found in Table 1. iNaturalist-2017 is the largest dataset for the fine-grained task. Compared with other datasets for this task, it contains 13 superclasses. Such a data distribution can provide a more convincing evaluation for the generalization ability of a model.

Baselines: We compared our method to the following baselines due to their state-of-the-art performance and high relevance. Note that for a fair comparison, we did not include methods using 1) additional data (from the web or other datasets), 2) human-annotated part locations and 3) hierarchical labels (i.e., species, genus, and family). And all of the compared methods in each table share the same backbone unless specified otherwise.

  • FCAN [21]: Fully convolutional attention network, which adaptively selects multiple attentions by reinforcement learning.

  • MDTP [31]: Mining discriminative triplets of patches, which utilize geometric constraints to improve the accuracy of patch localization.

  • DT-RAM [19]: Dynamic computational time model for recurrent visual attention, which attends on the most discriminative parts by dynamic steps.

  • SSN [22]: Saliency-based sampling networks, which conduct non-uniformed sampling based on saliency map in an end-to-end way.

  • MG-CNN [29]: Multiple granularity descriptors, which leverage the hierarchical labels to generate comprehensive descriptors.

  • STN [11]: Spatial transformer network, which conducts parameterized spatial transformation to obtain zoomed in or pose normalized objects.

  • RA-CNN [7]: Recurrent attention CNN, which recurrently attends on discriminative parts in multi-scale.

  • MA-CNN [40]: Multiple attention CNN, which attends on multiple parts by their proposed channel grouping module in a weakly-supervised way.

  • MAMC [26]: Multi-attention multi-class constraint network, which learns multiple attentions by conducting multi-class constraint over attended features.

  • NTSN [37]: Navigator-Teacher-Scrutinizer Network, which is a novel self-supervision mechanism to effectively localize informative regions without the need of bounding-box/part annotations.

  • iSQRT-COV [18]: Towards faster training of global covariance pooling networks by iterative matrix square root normalization.

Attention Description Accuracy

feature maps 83.5

trilinear attention 84.9

spacial norm 85.2

spacial norm 84.3

spacial norm 84.5

relation norm 85.0

spacial + relation 85.3
Table 2: Ablation experiments on attention module in terms of recognition accuracy on the CUB-200-2011 dataset.

Implementation: We used open-sourced MXNet [4] as our code-base, and trained all the models on 8 Tesla P-100 GPUs. The backbones are are pre-trained on Imagenet [23], and all of the performances are single-crop testing results for a single model unless specially stated. We used SGD optimizer without momentum and weight decay, and the batch size was set to 96. The temperature in Equation 6 is 10, and the loss weight in Equation 8 is 2. More implementation details can be referred to our code https://github.com/researchmm/tasn.

4.2 Evaluation and analysis on CUB-200-2011

Trilinear attention. Table 2 shows the impact of different normalization functions for the part-net in term of recognition accuracy. Specifically, we randomly select a channel of attention maps in each iteration in training stage, and conduct average pooling over attention maps for testing. All the models use Resnet-50 as the backbone with an input resolution of 224. It can be observed that trilinear attention maps can significantly outperform the original feature maps. Both the attention functions of and can improve the gain of trilinear attention. and bring a drop of performance, because such normalization functions is harmful for preserving spatial information. To this end, we adopt the last setting (of Table 2) in our TASN. Note that in the term , indicates the region that a channel is focusing on and denotes the feature of that region.

We further compared our trilinear attention module with “self-attention” [28]. Specifically, we followed [28] to obtain attention maps by , and the results show that the trilinear attention module can outperform self-attention module with 0.7% points increases.

Figure 5: A comparison of feature maps in (a) and trilinear attention maps in (b). Each column shows the same channel of feature maps and trilinear attention maps, and we randomly select nine channels for comparison. Compared to first-order feature maps, each channel of the trilinear attention maps focus on a specific part, without attending on background noises. [Best viewed in color]
Approach master-net TASN
Resnet-50 [8] 81.6 81.6
uniformed sampler 84.1 85.8
sampler in SSN [22] 84.8 85.3
our sampler 85.5 87.0
Table 3: Ablation experiments on sampling module in term of classification accuracy on the CUB-200-2011 dataset.
Resolution 224 280 336 392
Resnet-50 [8] 81.6 83.3 85.0 85.6
master-net 85.5 86.6 87.0 86.8
TASN 87.0 87.3 87.9 87.9
Table 4: Ablation experiments on distilling module with different input resolutions.

Attention-based sampler. To demonstrate the effectiveness of our attention-based sampling mechanism, we compared our sampling mechanism with 1) uniformed sampling (by binarizing the attention maps) and 2) sampling operation introduced in SSN [22]. We set the input attention maps to be same when comparing sampling mechanisms, and experiments were conducted on two cases, i.e., with and without part-net. All the models use Resnet-50 as the backbone and the input resolution is set to 224. The result in Table 3 shows that our sampling mechanism remarkably outperforms the baselines. SSN sampler obtains a better result than uniformed sampler without part-net, while the further improvements are limited when added part-net. These observations show that the spatial distortion caused by SSN sampler is harmful for preserving subtle details.

Knowledge distilling. Table 4 reveals the impact of details distilling module with different input resolutions. We can observe consistency improvements by details distilling. The performance of Resnet-50 [8] is saturated with 85.6%, and 448 input can not further improve the accuracy. Without distiller (i.e., master-net only), the performance is slightly dropped with 392 input (compared to 336 input), since it is difficult to optimize each detail with large feature resolutions (a similar drop can also be observed on Resnet-50 with 672 inputs).

Moreover, to study the attention selection strategy (i.e., ranking selection vs. random selection), we ranked attention maps by their response, and sample high response ones with large possibility, while the recognition performance dropped from 87.0% to 86.8%. The reason is that ranking makes some parts rarely picked, while such parts can also benefit details learning. We also conducted experiments on distilling two parts in each iteration, and the result is the same as distilling one part each time.

Compared to sampling-based methods. We compare our TASN with three sampling-based methods: 1) uniformed sampling with high resolution (i.e., zoom in), 2) uniformed sampling with attention (i.e., crop) and 3) non-uniformed sampling proposed in SSN [22]. As shown in Table 5, higher resolution can significantly improve fine-grained recognition performance by 4.9% relatively. However, 448 input increases the computational cost (i.e., flops) by four times compared to 224 input. SSN [22] obtains a better results than DT-RAM [19], and our TASN can further obtain 2.9% relative improvement. Such improvements mainly come from two aspects: 1) a better sampling mechanism considering spatial distortion (1.2%), and 2) a better fine-grained details optimizing strategy (1.7%).

Approach Resolution Accuracy
Resnet-50 [8] 224 81.6
Resnet-50 [8] 448 85.6
DT-RAM [19] 224 82.8
SSN [22] 227 84.5
TASN (ours) 224 87.0
Table 5: Comparison with sampling-based methods in terms of classification accuracy on the CUB-200-2011 dataset.
Approach Backbone Accuracy
MG-CNN [29] 3VGG-16 81.7
ST-CNN [11] 3Inception-v2 84.1
RA-CNN [7] 3VGG-19 85.3
MA-CNN [40] 3VGG-19 85.4
TASN (ours) 1VGG-19 86.1
TASN (ours) 3VGG-19 87.1
MAMC [26] 1Resnet-50 86.5
NTSN [37] 3Resnet-50 87.3
TASN (ours) 1Resnet-50 87.9
Table 6: Comparison with part-based methods (all the results are reported in high-resolution setting) in terms of classification accuracy on the CUB-200-2011 dataset.

Compared to attention-based part methods. In Table 6, we compare our TASN to attention-based parts methods. For a fair comparison, 1) high-resolution input is adopted by all methods and 2) the same backbone numbers are used. It can be observed that for VGG based methods, our TASN outperforms all the baselines even with only one backbone. Moreover, after ensembling three backbones (trained with different parameter settings), TASN can improve the performance by 1.9% over the best 3 parts model MA-CNN [40]. Moreover, our 3 streams result can also outperform 6 streams MA-CNN (86.5%) with a margin of 0.7%. We do not ensemble more streams as the model ensemble is beyond this work. For Resnet-50 based method: compared with the state-of-the-art single-stream MAMC [26], our TASN achieves a remarkable improvement by 1.6%. Moreover, although NTSN [37] () concatenates global feature with two part features, our single-stream TASN still can achieve 0.6% points increases.

Combining with second-order feature learning methods. In Table 7, we exhibit that our TASN learns a strong first-order representation, which can further improve the performance of second-order feature methods. Specifically, compared to the best second-order methods iSQRT-COV [18], our TASN 2k first-order feature outperforms their 8k feature with an improvement by 0.7%, which shows the effectiveness of our TASN. Moreover, we transfer their released code to our framework and obtain an accuracy of 89.1%, which shows the compatibility of these two methods. Note that for a fair comparison, we follow their settings and predict the label of a test image by averaging prediction scores of the image and its horizontal flip.

Approach Dimension Accuracy
iSQRT-COV [18] 8k 87.3
iSQRT-COV [18] 32k 88.1
TASN (ours) 2k 87.9
TASN + iSQRT-COV 32k 89.1
Table 7: Extensive experiments on combining second-order feature learning methods.

4.3 Evaluation and analysis on Stanford-Car

Approach Backbone Accuracy
Baseline 1VGG-19 88.6
master-net 1VGG-19 90.3
TASN 1VGG-19 92.4
TASN (ensemble) 2VGG-19 93.1
TASN (ensemble) 3VGG-19 93.2
Table 8: Component analysis in terms of classification accuracy on the Stanford-Car dataset.
Approach Backbone Accuracy
FCAN [21] 3VGG-16 91.3
MDTP [31] 3VGG-16 92.5
RA-CNN [7] 3VGG-19 92.5
MA-CNN [40] 3VGG-19 92.6
TASN (ours) 1VGG-19 92.4
TASN (ours) 3VGG-19 93.2
MAMC [26] 1Resnet-50 92.8
NTSN [37] 3Resnet-50 93.7
TASN (ours) 1Resnet-50 93.8
Table 9: Comparison in terms of classification accuracy on the Stanford-Car dataset.

Table 8 shows the result of VGG-19 baseline, our master-net, a single TASN model, and TASN ensemble results. We can observe 1.9% relative improvements by structure preserved sampling and further improvements of 2.3% by the full model. Table 9 compares TASN with attention-based parts methods. Specifically, TASN with single VGG-19 achieves comparable results with 3 streams part methods. And our ensembled 3 streams TASN outperforms the best 3 streams part learning methods MA-CNN [40]. Compared to their 5 streams result (92.8%), our result is still better. For Resnet-50 based method, we compare our TASN to the state-of-the-art method MAMC [26], and achieve 1.1% improvements. Moreover, our single-stream TASN can achieve slightly better performance than NTSN [37], which concatenates a global feature with two part features.

4.4 Evaluation and analysis on iNaturalist 2017

Super Class # Class Resnet [8] SSN [22] TASN
Plantae 2101 60.3 63.9 66.6
Insecta 1021 69.1 74.7 77.6
Aves 964 59.1 68.2 72.0
Reptilia 289 37.4 43.9 46.4
Mammalia 186 50.2 55.3 57.7
Fungi 121 62.5 64.2 70.3
Amphibia 115 41.8 50.2 51.6
Mollusca 93 56.9 61.5 64.7
Animalia 77 64.8 67.8 71.0
Arachnida 56 64.8 73.8 75.1
Actinopterygii 53 57.0 60.3 65.5
Chromista 9 57.6 57.6 62.5
Protozoa 4 78.1 79.5 79.5
Total 5089 59.6 65.2 68.2
Table 10: Comparison in terms of classification accuracy on the iNaturalist 2017 dataset.

We also conduct our TASN on the largest fine-grained dataset, i.e., iNaturalist 2017. We compare to Resnet [8] baseline and the best sampling method SSN [22]. All the models use Resnet-101 as the backbone with an input resolution of 224. As there are 13 superclasses in this dataset, we re-implement SSN [22] with their released code to obtain the performance on each superclass. The results are shown in Table 10, and we can observe that TASN outperforms Resnet baseline and SSN on every superclass. It is notable that compared to Resnet-101, TASN significantly improves the performance, especially on Reptilia (improved by 24.0%, relatively) and Aves (improved by 21.8%, relatively), which indicates that such superclasses contain more fine-grained details.

5 Conclusion

In this paper, we proposed a trilinear attention sampling network for fine-grained image recognition, which can learn rich feature representations from hundreds of part proposals. Instead of ensembling multiple part CNNs, we adopted knowledge distilling method to integrate fine-grained features into a single stream, which is not only efficient but also effective. Extensive experiments in CUB-Bird, iNaturalist 2017 and Stanford-Car demonstrate that TASN is able to outperform part-ensemble models even with a single stream. In the future, we will further study the proposed TASN in the following directions: 1) attention selection strategy, i.e., learning to select which details should be learned and distilled instead of randomly selecting, 2) conduct attention-based sampling over convolutional features instead of only over images, and 3) extend our work to other vision tasks, e.g., object detection and segmentation.

Acknowledgement: This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants 61622211 and 61620106009 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.


  • [1] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In CVPR, pages 2011–2018, 2014.
  • [2] Steve Branson, Grant Van Horn, Serge J. Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. In BMVC, 2014.
  • [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834–848, 2018.
  • [4] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015.
  • [5] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In CVPR, pages 4109–4118, 2018.
  • [6] Luc Devroye. Sample-based non-uniform random variate generation. In WSC, pages 260–265. ACM, 1986.
  • [7] Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In CVPR, pages 4438–4446, 2017.
  • [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  • [9] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adversarial samples supporting decision boundary. CoRR, abs/1805.05532, 2018.
  • [10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. stat, 1050:9, 2015.
  • [11] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In NIPS, pages 2017–2025, 2015.
  • [12] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In NIPS, pages 1571–1581, 2018.
  • [13] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In ICCV Workshop, 2013.
  • [14] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 2014.
  • [15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106–1114, 2012.
  • [16] Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. Fine-grained recognition as hsnet search for informative image parts. In CVPR, pages 6497–6506. IEEE, 2017.
  • [17] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  • [18] Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In CVPR, pages 947–955, 2018.
  • [19] Zhichao Li, Yi Yang, Xiao Liu, Feng Zhou, Shilei Wen, and Wei Xu. Dynamic computational time for visual attention. In ICCV, pages 1199–1209, 2017.
  • [20] Xinchen Liu, Wu Liu, Huadong Ma, and Huiyuan Fu. Large-scale vehicle re-identification in urban surveillance videos. In ICME, pages 1–6. IEEE, 2016.
  • [21] Xiao Liu, Tian Xia, Jiang Wang, Yi Yang, Feng Zhou, and Yuanqing Lin. Fully convolutional attention networks for fine-grained recognition. arXiv preprint arXiv:1603.06765, 2016.
  • [22] Adria Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba. Learning to zoom: a saliency-based sampling layer for neural networks. In ECCV, pages 51–66, 2018.
  • [23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211–252, 2015.
  • [24] Marcel Simon and Erik Rodner. Neural activation constellations: Unsupervised part model discovery with convolutional networks. In ICCV, pages 1143–1151, 2015.
  • [25] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, pages 1409–1556, 2015.
  • [26] Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. Multi-attention multi-class constraint for fine-grained image recognition. In ECCV, pages 805–821, 2018.
  • [27] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. 2018.
  • [28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017.
  • [29] Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. Multiple granularity descriptors for fine-grained categorization. In ICCV, pages 2399–2406, 2015.
  • [30] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794–7803, 2018.
  • [31] Yaming Wang, Jonghyun Choi, Vlad Morariu, and Larry S Davis. Mining discriminative triplets of patches for fine-grained classification. In CVPR, pages 1163–1172, 2016.
  • [32] Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. Selective convolutional descriptor aggregation for fine-grained image retrieval. TIP, 26(6):2868–2881, 2017.
  • [33] Xiu-Shen Wei, Chen-Wei Xie, Jianxin Wu, and Chunhua Shen. Mask-cnn: Localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recognition, 76:704–714, 2018.
  • [34] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
  • [35] Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In CVPR, pages 842–850, 2015.
  • [36] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine-grained categorization and verification. In CVPR, pages 3973–3981, 2015.
  • [37] Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. Learning to navigate for fine-grained classification. In ECCV, pages 420–435, 2018.
  • [38] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, pages 4133–4141, 2017.
  • [39] Xiaopeng Zhang, Hongkai Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. Picking deep filter responses for fine-grained image recognition. In CVPR, pages 1134–1142, 2016.
  • [40] Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. In ICCV, pages 5209–5217, 2017.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description