SCAttNet: Semantic Segmentation Network with Spatial and Channel Attention Mechanism for High-Resolution Remote Sensing Images
High-resolution remote sensing images (HRRSIs) contain substantial ground object information, such as texture, shape, and spatial location. Semantic segmentation, which is an important method for element extraction, has been widely used in processing mass HRRSIs. However, HRRSIs often exhibit large intraclass variance and small interclass variance due to the diversity and complexity of ground objects, thereby bringing great challenges to a semantic segmentation task. In this study, we propose a new end-to-end semantic segmentation network, which integrates two lightweight attention mechanisms that can refine features adaptively. We compare our method with several previous advanced networks on the ISPRS Vaihingen and Potsdam datasets. Experimental results show that our method can achieve better semantic segmentation results compared with other works. The source codes are available at https://github.com/lehaifeng/SCAttNet.
The Semantic segmentation of remote sensing images is a fundamental task that classifies each pixel in an image into a specified category. It plays an important role in many fields such as change detection, element extraction, and military target recognition.
Image semantic segmentation methods can be divided into two categories: traditional methods and deep learning based ones. Traditional methods use the color, texture, shape, and spatial position relationships of an object to extract features and then use clustering, classification, and threshold algorithms to segment an image . However, these methods depend heavily on artificial design features and show some bottlenecks. Recently, deep learning based methods have been regarded as a promising approach to solve image semantic segmentation problems . For example, methods based on fully convolutional network (FCN)  have achieved state-of-the-art segmentation results on many natural image datasets, such as PASCAL VOC  and Cityscapes .
However, remote sensing images are different from natural images and often viewed from a high-altitude angle. Thus, the range of imaging is wide, and the background is complex and diverse. Especially in HRRSIs, the difference of ground objects becomes further notable. To segment HRRSIs effectively, Chen et al.  used a shuffle convolution neural network and found that the method is effective in segmenting small objects. Liu et al.  designed an hourglass-shaped network (HSN) network based on inception network and skip connection and the network achieved better results compared with the reference networks on the ISPRS Vaihingen and ISPRS Potsdam datasets. Guo et al.  learned from the spatial pyramid pooling model to capture multiscale features in HRRSIs, and then used conditional random fields to perform postclassification refinement. Kampffmeyer et al. used a weighted loss function to solve the problem of category imbalance. Chen et al.  designed two deep networks based on residual module, and achieved better results compared with FCN-8s  and SegNet . In summary, these networks mainly change the depth and width of the network, or combine multimodal data such as digital surface models (DSMs) to improve the accuracy of the network.
Recently, attention mechanisms have been successfully applied to semantic segmentation. These methods can be divided into two categories. One uses attention mechanism to select meaningful features at channel axis. For example, PAN uses a global attention mechanism as a context prior to select precise features channel-wisely. Attention U-net uses a channel attention mechanism to control the fusion of high-level and low-level features at channel axis. However, these attention mechanisms do not consider to enhance the feature representation at spatial axis. Another one is called the self attention mechanism, which calculates the feature representation in each position by weighted sum the features of all other positions. Thus, it can model the long-range context information for semantic segmentation task. For example, DANet uses two self-attention mechanisms to model long-range context information at channel and spatial axes respectively. However, these methods are complexity in the size of model, thus yield inefficient computation which will be great challenge to process massive remote sensing images. In our work, two lightweight attention mechanisms  which contains spatial attention and channel attention are adopted for the semantic segmentation of HRRSIs. The spatial attention mechanism decides where to enhance, and the channel attention decides what to enhance. Integrating these two attention mechanisms can effectively improve the accuracy of semantic segmentation. The contributions of this study primarily include the following three points:
(1) A new semantic segmentation network that integrates a spatial attention mechanism and a channel attention mechanism are proposed to improve the semantic segmentation accuracy of HRRSIs.
(2) The working principle of the attention mechanism is analyzed through visualization to explain why our method works.
(3) Experiments reveal that the attention mechanism can focus on valuable areas and thus our method shows remarkable performance on small objects.
Ii-a Overview of SCAttNet
The proposed semantic segmentation network is shown in Fig. 1. It consists of two parts: a backbone for feature extraction and an attention module. The attention module is composed of a channel attention and a spatial attention in cascade. For an input remote sensing image, we first use the backbone network for feature extraction. Then we feed the extracted feature map into the channel attention mechanism to refine the features in channels. Afterward, we feed the refined channel feature map into the spatial attention model for refinement in space. Lastly, we can obtain the semantic segmentation results via convolution and SoftMax operations. Specific network details and design ideas are shown in the following sections.
Ii-B Attention Mechanism
Channel Attention: Given a high-resolution remote sensing image, it will produce a multichannel feature map (where C, H, and W denote the number of channels, the height, the width of the feature map, respectively) after passing through several convolutional layers. The information expressed in the feature map of each channel is different. Channel attention aims to use the relationships between each channel of the feature map to learn a 1D weight , and then multiply it to the corresponding channel. In this manner, it can pay more attention to the meaningful semantic information for the current task. To learn effective weight representation, we first aggregate spatial dimension information through global average pooling and global max pooling to generate two feature descriptors for each channel. Then we feed the two feature descriptors into a shared multilayer perceptron with one hidden layer (where the number of the hidden layer units is C/8) to generate more representative feature vectors. Afterward, we merge the output feature vectors through an element-wise summation operation. Finally, using a sigmoid function, we can obtain the final channel attention map. The flow chart is illustrated in the channel attention module of Fig. 1. The formula for calculating channel attention is shown in Formula 1:
Spatial Attention: For a spatial attention, it focuses on where are valuable for current tasks. In HRRSIs, the ground objects are exhibited in various sizes and the distribution is complicated. Therefore, using spatial attention is useful for aggregating spatial information, especially for small ground objects. Spatial attention utilizes the relationships between different spatial positions to learn a 2D spatial weight map and then multiplies it to the corresponding spatial position to learn more representative features. To learn the spatial weight relationships effectively, we first generate two feature descriptors for each spatial position through global average pooling and global max pooling operations. Then we concentrate two feature descriptors together and generate a spatial attention map through a convolution operation. Lastly, we use a sigmoid function to scale the spatial attention map to . The flow chart is illustrated in the spatial attention module of Fig. 1. The spatial attention calculation formula is shown in Formula 2.
where represents a convolution operation with kernel size.
In this study, we follow the method of Woo to integrate the two attention mechanisms. First, we use channel attention to capture good semantic information, and then use spatial attention to capture spatial location information, which is conducive to extracting more valuable feature expression.
Ii-C Backbone for feature extraction
In this study, we use two representative backbones for feature extraction: SegNet and ResNet50. On this basis, we proposed two networks: SCAttNet V1 with SegNet as backbone and SCAttNet V2 with ResNet50 as backbone. For SegNet network, it has been widely used as the baseline model of remote sensing images semantic segmentation and achieved sound semantic segmentation results. For example, Audebert et al.  based on SegNet network, combined with multi-modal data, achieved state-of-the-art semantic segmentation results on the ISPRS Vaihingen dataset. And ResNet50 is also a common backbone for semantic segmentation tasks because it can build a deep-layer model with a wide receptive field. We use eight times downsampling for ResNet50 in our work. Unlike the SegNet and ResNet50 networks, we do not directly use the features of the last layer for semantic inference, instead, we feed the feature map of the last layer into the attention module for feature refinement and then make semantic inference, which is conducive to learning better feature expression.
We can easily compute that the number of parameters of the attention modules are so small that they can be neglected for both our SCAttNet V1 and SCAttNet V2, which have tens of millions of parameters. Specifically, the number of parameters of the channel attention mechanism is (C is the number of channels) and the and the number of parameters of the spatial attention mechanism is 98. In addition, we only add the attention mechanism at the end layer of the backbone network; thus, it seldom brings computation complexity to our proposed network.
|Model||Imp. Surfaces||Building||Low veg.||Tree||Car||MIoU (%)||AFÂ (%)||OA(%)|
|SegNet + cha. att||76.74/86.84||80.46/89.17||63.31/77.53||65.13/78.89||41.67/58.83||65.46||78.25||83.48|
|SegNet + spa. att||74.46/85.36||77.01/87.01||63.37/77.58||65.08/78.85||40.30/57.45||64.05||77.25||82.38|
|Model||Imp. Surfaces||Building||Low Veg.||Tree||Car||Clutter/background||MIoU(%)||AF (%)||OA(%)|
In this section, we evaluate the performance of our network on the ISPRS Vaihingen and ISPRS Potsdam datasets
Iii-a Datasets and evaluation metrics
The ISPRS Vaihingen dataset contains 33 orthophoto maps and related DSMs. Sixteen of them are labeled. The average size of the images is and the resolution is 9 cm. Each image contains three bands: near infrared, red and green bands. Moreover, it includes six categories: impervious surfaces, buildings, low vegetation, trees, cars and clutter/background. The ISPRS Potsdam dataset contains 38 orthophoto maps and related normalized DSMs. Twenty-four of them are labeled. The size of each image is and the resolution is 5 cm. Each image contains four bands: near infrared, red, green and blue bands. It has the similar categories as the ISPRS Vaihingen dataset.
To evaluate our proposed model, we use three evaluation metrics including mean inter-section over union (MIoU), average F1-score (AF) and overall accuracy (OA) to evaluate semantic segmentation performance.
Iii-B Implementation details
Considering that many remote sensing datasets in practice do not have DSMs, we do not use such datasets in this experiment for wide application value. For the ISPRS Vaihingen dataset, we divide the labeled dataset into two parts, in which (ID 30, 32, 34, 37) are for evaluating the performance of the network, and the remaining 12 images are for training. The number of Vaihingen dataset is small; thus, to prevent overfitting, we first crop the training dataset randomly into a size and then expand the data through rotation and translation operations, and finally obtain 12,000 patches for training. The Potsdam dataset is also divided into two parts, in which (ID 2_12, 3_12, 4_12, 5_12, 6_12, 7_12) are for testing, and the remaining 18 images are for training. Then, we crop it randomly and obtain 27,000 patches for training. We trained all the model from the scratch without belts and whistles. The models including FCN-32s, FCN-8s, U-net, SegNet and G-FRNet all use VGG-16 as backbone. As for RefineNet and DeepLabv3+, we adopt the ResNet50 as backbone with 32 times downsampling for RefineNet and eight times downsampling for DeepLabv3+ as the original paper. We also adopt ResNet50 as backbone with eight times downsampling for CBAM. To train the proposed SCAttNet v1, we set the learning rate of SCAttNet V1 on the Vaihingen and Potsdam datasets to 1e-3 and 1e-4, respectively. To train SCAttNet V2, we set the learning rate of SCAttNet V2 on both datasets to 1e-3. The proposed models all adopt Adam as the optimizer and cross-entropy as the loss function. And the training epochs is set to 50. Considering limited computing resources, the batch size is set to 16. To test all the above models, we only use a sliding window without overlap to crop the images and then stitch together. We conduct all our experiments in Tensorflow platform with a NVIDIA 1080Ti GPU.
Iii-C Vaihingen dataset
Table LABEL:tabel1 reports the semantic segmentation of the ISPRS Vaihingen dataset. We adopt the practice of  and do not report the accuracy of the clutter/background class because the Vaihingen dataset has less clutter/background. From the experimental results in Table 1, the MIoU/AF/OA increased by 1.4%/1.25%/0.56% after incorporating the channel attention module, compared with original SegNet. And it achieves competitive results as the SegNet network after incorporating spatial attention module, but the performance has an obvious increase on small objects such as cars. With the combination of channel and spatial attention mechanisms, the MIoU and AF increased by 2.9% and 2.5% compared with the original SegNet. Moreover, in SCAttNet V2, the MIoU/AF/OA increased by 1.21%/0.83%/0.90% compared with ResNet50 which as a baseline model. In addition, we find that in FCN-32s, the car category shows a low accuracy, but the car category in FCN-8s has improved much compared with FCN-32s, which shows the importance of low-level features for segmenting small objects.
To compare the semantic segmentation results, we visualize the semantic segmentation results of ID32. The visualization results are shown in the first row of Fig. 2. We can see that our SCAttNet V1 achieves more coherent results compared with the original SegNet model. In addition, our model infers more accurately in small object areas such as cars in the right middle of the image. We also test the accuracy of the ID32 area. The MIoU/AF /OA of SegNet is 60.57%/74.01%/82.78%, whereas that of our SCAttNet V1 is 64.92%/77.74%/85.49%. On this basis, our network can improve semantic segmentation accuracy by incorporating the attention mechanism effectively.
Iii-D Potsdam dataset
Table II reports the semantic segmentation results of the ISPRS Potsdam dataset. From the semantic segmentation results, the proposed SCAttNet V2 model is superior to the comparative models. Moreover, it achieves 1.12%/1.08%/1.03% higher in MIoU/AF/OA metrics compared with ResNet50 which as a baseline model. In SCAttNet V1, The MIoU/AF/OA has increased by 2.08%, 1.74% and 1.35%, respectively compared with SegNet which as a baseline model. Thus, we further confirm the effectiveness of the attention mechanism.
The second row of Fig. 2 shows the visualization results of the ID 5_12 area of the ISPRS Potsdam dataset. Compared with the comparative model, our proposed network achieves improved segmentation results in the building area, especially in terms of avoiding background interference. We also test the semantic segmentation performance in this area. The MIoU/AF/OA of the SegNet network is 57.19%/68.87%/83.81%, whereas that of SCAttNet V1 network is 59.50%/71.16%/85.04%. We find that our network achieves higher accuracy in all three metrics compared with the comparative model.
Iii-E Quantitative analysis of improved interpretability
To analyze the enhancement of attention mechanism to the network, we visualize the feature expression of the SegNet network and our SCAttNet V1 network on the ISPRS Vaihingen dataset. We can simply overlay the heatmap of the network before SoftMax operation with an original image, and the relevant areas can be highlighted for a specific category. The visualization results are shown in Fig. 3. After incorporating the attention mechanism, our network can focus on the areas of target and suppress the influence of other categories better than before. As shown in the first column of Fig.3, after incorporating the attention mechanism, our network only has a strong response in the car category and other areas are cold toned, which means that the relevance in other areas is very low. However, in the original SegNet network, the impervious area also shows a certain response that may cause interference for the car category. In addition, as shown in the building category in the third column of Fig.3 , the impervious surface area is suppressed after incorporating the attention mechanism. Thus, the impervious surface area avoids classifying into buildings.
In this study, we propose a new semantic segmentation network that can adaptively refine features based on the attention mechanism. Experiments on the ISPRS Vaihingen and Potsdam datasets demonstrate the effectiveness of our method. However, some shortcomings remain. The study only uses two common attention mechanisms. Thus, how to design the attention mechanism effectively and capture more discriminative features for semantic inference remains a promising direction for future work.
This work was supported by the National Natural Science Foundation of China [grant numbers 41571397, 41501442, and 51678077].
- ISPRS 2d semantic labeling dataset.http://www2.isprs.org/ commissions/comm3/wg4/semantic-labeling.html
- (2017) Gated feedback refinement network for dense image labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3751–3759. Cited by: TABLE I, TABLE II.
- (2016) Semantic segmentation of earth observation data using multimodal and multi-scale deep networks. In Asian Conference on Computer Vision, pp. 180–196. Cited by: §II-C.
- (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39 (12), pp. 2481–2495. Cited by: §I, §I, TABLE I, TABLE II.
- (2018) Symmetrical dense-shortcut deep fully convolutional networks for semantic segmentation of very-high-resolution remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11 (5), pp. 1633–1644. Cited by: §I.
- (2018) Semantic segmentation of aerial images with shuffling convolutional neural networks. IEEE Geoscience and Remote Sensing Letters 15 (2), pp. 173–177. Cited by: §I.
- (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801–818. Cited by: TABLE I, TABLE II.
- (2016) The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223. Cited by: §I.
- (2010) The pascal visual object classes (voc) challenge. International journal of computer vision 88 (2), pp. 303–338. Cited by: §I.
- (2019) Dual attention network for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3146–3154. Cited by: §I.
- (2018) Pixel-wise classification method for high resolution remote sensing imagery using deep neural networks. ISPRS International Journal of Geo-Information 7 (3), pp. 110. Cited by: §I, §III-C.
- (2016) Identity mappings in deep residual networks. In European conference on computer vision, pp. 630–645. Cited by: §II-C.
- (2011) Information fusion of aerial images and lidar data in urban areas: vector-stacking, re-classification and post-processing approaches. International Journal of Remote Sensing 32 (1), pp. 69–84. Cited by: §I.
- (2019) Ccnet: criss-cross attention for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 603–612. Cited by: §I.
- (2016) Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 1–9. Cited by: §I, §III-C.
- (2018) Pyramid attention network for semantic segmentation. arXiv preprint arXiv:1805.10180. Cited by: §I.
- (2017) Refinenet: multi-path refinement networks for high-resolution semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1925–1934. Cited by: TABLE I, TABLE II.
- (2017) Hourglass-shapenetwork based semantic segmentation for high resolution aerial imagery. Remote Sensing 9 (6), pp. 522. Cited by: §I, §III-C.
- (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440. Cited by: §I, §I, TABLE I, TABLE II.
- (2018) Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999. Cited by: §I.
- (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Cited by: §I, TABLE I, TABLE II.
- (2018) Cbam: convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19. Cited by: §I, §II-B, TABLE I, TABLE II.
- (2012) Layered object models for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (9), pp. 1731–1743. Cited by: §I.