Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks
Fully convolutional neural networks (F-CNNs) have set the state-of-the-art in image segmentation for a plethora of applications. Architectural innovations within F-CNNs have mainly focused on improving spatial encoding or network connectivity to aid gradient flow. In this paper, we explore an alternate direction of recalibrating the feature maps adaptively, to boost meaningful features, while suppressing weak ones. We draw inspiration from the recently proposed squeeze & excitation (SE) module for channel recalibration of feature maps for image classification. Towards this end, we introduce three variants of SE modules for image segmentation, (i) squeezing spatially and exciting channel-wise (cSE), (ii) squeezing channel-wise and exciting spatially (sSE) and (iii) concurrent spatial and channel squeeze & excitation (scSE). We effectively incorporate these SE modules within three different state-of-the-art F-CNNs (DenseNet, SD-Net, U-Net) and observe consistent improvement of performance across all architectures, while minimally effecting model complexity. Evaluations are performed on two challenging applications: whole brain segmentation on MRI scans (Multi-Atlas Labelling Challenge Dataset) and organ segmentation on whole body contrast enhanced CT scans (Visceral Dataset).
Deep learning, in particular, convolutional neural networks (CNN) have become the standard for image classification [1, 2]. Fully convolutional neural networks (F-CNNs) have become the tool of choice for many image segmentation tasks in medical imaging [3, 4, 5] and computer vision [6, 7, 8, 9]. The basic building block for all these architectures is the convolution layer, which learns filters capturing local spatial pattern along all the input channels and generates feature maps jointly encoding the spatial and channel information. While much effort is put into improving this joint encoding of spatial and channel information, encoding of the spatial and channel-wise patterns independently is less explored. Recent work attempted to address this issue by explicitly modeling the interdependencies between the channels of feature maps. An architectural component called squeeze & excitation (SE) block  was introduced, which can be seamlessly integrated within any CNN model. The SE block factors out the spatial dependency by global average pooling to learn a channel specific descriptor, which is used to recalibrate the feature map to emphasize on useful channels. Its nomenclature is motivated by the fact that the SE block ‘squeezes’ along the spatial domain and ‘excites’ or reweights along the channels. A convolutional network with SE blocks won the first place in the ILSVRC 2017 classification competition on the ImageNet dataset, indicating its effectiveness .
In this work, we want to leverage the high performance of SE blocks for image classification to image segmentation with F-CNNs. We refer to the previously introduced SE block as channel SE (cSE), because it only excites channel-wise, which proved to be effective for classification. We hypothesize that for image segmentation, the pixel-wise spatial information is more informative. Hence, we introduce another SE block, which ‘squeezes’ along the channels and ‘excites’ spatially, termed spatial SE (sSE). Finally, we propose to have concurrent spatial and channel SE blocks (scSE) that recalibrate the feature maps separately along channel and space, and then combines the output. Encouraging feature maps to be more informative both spatially and channel-wise. To the best of our knowledge, this is the first time that spatial squeeze & excitation is proposed for neural networks and the first integration of squeeze & excitation in F-CNNs.
We integrate the proposed SE blocks (cSE, sSE and scSE) in three state-of-the-art F-CNN models for image segmentation to demonstrate that SE blocks are a generic network component to boost performance. We evaluate the segmentation performance in two important medical applications: whole-brain and whole-body segmentation. In whole-brain segmentation, we automatically identify 27 cortical and subcortical structures on magnetic resonance imaging (MRI) T1-weighted brain scans. In whole-body segmentation, we automatically label 10 visceral organs on contrast-enhanced CT scan of the abdomen.
Related Work: F-CNN architectures have successfully been used in a wide variety of medical image segmentation tasks to provide state-of-the-art performance. A seminal F-CNN model is U-Net , which has an encoder/decoder based architecture combined with skip connections between encoder and decoder blocks with similar spatial resolution. SkipDeconv-Net (SD-Net)  builds upon U-Net, using unpooling layers used in  for decoding, learnt by jointly optimizing logistic and Dice loss functions. A more recent architecture introduces dense connectivity within the encoder and decoder blocks, unlike U-Net and SD-Net which uses normal convolutions, termed fully convolutional DenseNet .
Let us assume an input feature map that passes through an encoder or decoder block to generate output feature map , . Here and are the spatial height and width, with and being the input and output channels, respectively. The generated combines the spatial and channel information of through a series of convolutional layers and non-linearities defined by . We place the SE blocks on to recalibrate it to . We propose three different variants of SE blocks, which are detailed next. The SE blocks can be seamlessly integrated within any F-CNN model by placing them after every encoder and decoder block, as illustrated in Fig. 1(a). is used in the subsequent pooling/upsampling layers.
2.1 Spatial Squeeze and Channel Excitation Block (cSE)
We describe the spatial squeeze and channel excitation block, which was proposed in . We consider the input feature map as a combination of channels . Spatial squeeze is performed by a global average pooling layer, producing vector with its element
This operation embeds the global spatial information in vector . This vector is transformed to , with , being weights of two fully-connected layers and the ReLU operator . This encodes the channel-wise dependencies. The dynamic range of the activations of are brought to the interval , passing it through a sigmoid layer . The resultant vector is used to recalibrate or excite to
The activation indicates the importance of the channel, which is either scaled up or down. As the network learns, these activations are adaptively tuned to ignore less important channels and emphasize the important ones. The architecture of the block is illustrated in Fig. 1(b).
2.2 Channel Squeeze and Spatial Excitation Block (sSE)
We introduce the channel squeeze and spatial excitation block that squeezes the feature map along the channels and excites spatially, which we consider important for fine-grained image segmentation. Here, we consider an alternative slicing of the input tensor , where corresponding to the spatial location with and . The spatial squeeze operation is achieved through a convolution with weight , generating a projection tensor . Each of the projection represents the linearly combined representation for all channels for a spatial location . This projection is passed through a sigmoid layer to rescale activations to , which is used to recalibrate or excite spatially
Each value corresponds to the relative importance of a spatial information of a given feature map. This recalibration provides more importance to relevant spatial locations and ignores irrelevant ones. The architectural flow is shown in Fig. 1(c).
2.3 Spatial and Channel Squeeze & Excitation Block (scSE)
Finally, we introduce a combination of the above two SE blocks, which concurrently recalibrates the input spatially and channel-wise. We obtain the concurrent spatial and channel SE, , by element-wise addition of the channel and spatial excitation, . A location of the input feature map is given higher activation when it gets high importance from both, channel re-scaling and spatial re-scaling. This recalibration encourages the network to learn more meaningful feature maps, that are relevant both spatially and channel-wise. The architecture of the combined scSE block is illustrated in Fig. 1(d).
2.4 Model Complexity
Let us consider an encoder/decoder block, with an output feature map of channels. Addition of a cSE block introduces new weights, while a sSE block introduces weights. So, the increase in model complexity of a F-CNN with encoder/decoder blocks is , where is the number of output channels for the encoder/decoder block. To give a concrete example, the U-Net in our experiments has about parameters. The scSE block adds parameters, which is an approximate increase by 1.5%. Hence, SE blocks only increase overall network complexity by a very small fraction.
3 Experimental Results
In this section, we conducted extensive experiments to explore the impact of our proposed modules. We chose three state-of-the-art F-CNN architectures, U-Net , SD-Net  and Fully Convolutional DenseNet . All of the networks have an encoder/decoder based architecture. The encoding and decoding paths consist of repeating blocks separated by down-sampling and up-sampling, respectively. We insert (i) channel-wise SE (cSE) blocks, (ii) spatial SE (sSE) blocks and (iii) concurrent spatial and channel-wise SE (scSE) blocks after every encoder and decoder block of the F-CNN architecture and compare against its vanilla version.
Datatsets: We use two datasets in our experiments. (i) Firstly, we tackle the task of segmenting MRI T1 brain scans into 27 cortical and sub-cortical structures. We use the Multi-Atlas Labelling Challenge (MALC) dataset , which is a part of OASIS , with 15 scans for training and 15 scans for testing consistent to the challenge instructions. The main challenge associated with the dataset are the limited training data with severe class imbalance between the target structures. Manual segmentations for MALC were provided by Neuromorphometrics, Inc.***http://Neuromorphometrics.com/ (ii) Secondly, we tackle the task of segmenting 10 organs on whole-body contrast enhanced CT (ceCT) scans. We use data from the Visceral dataset . We train on 65 scans from the silver corpus, and test on 20 scans with manual annotations from the gold corpus. The silver corpus was automatically labeled by fusing the results of multiple algorithms, yielding noisy labels. The main challenge associated with the whole-body segmentation is the highly variable shape of the visceral organs and the capability to generalize when trained with noisy labels. We use Dice score for performance evaluation.
Model Learning: In our experiments, all of the three F-CNN architectures had 4 encoder blocks, one bottleneck layer, 4 decoder blocks and a classification layer at the end. The logistic loss function was weighted with median frequency balancing  to compensate for the class imbalance. The learning rate was initially set to and decreased by one order after every 10 epochs. The momentum was set to , weight decay constant to and a mini batch size of . Optimization was performed using stochastic gradient descent. Training was continued till validation loss converged. All the experiments were conducted on an NVIDIA Titan Xp GPU with 12GB RAM.
|Networks||No SE Block||+ cSE Block||+ sSE Block||+ scSE Block|
|Networks||No SE Block||+ cSE Block||+ sSE Block||+ scSE Block|
Quantitative Results: Table 1 lists the mean Dice score on test data for both datasets. Results of the standard networks together with the addition of cSE, sSE and scSE blocks are reported. Comparing along the columns, we observe that inclusion of any SE block consistently provides a statistically significant (, Wilcoxon signed-rank) increase in Dice score in comparison to the normal version for all networks, in both applications. We further observe that the spatial excitation yields a higher increase than the channel-wise excitation, which confirms our hypothesis that spatial excitation is more important for segmentation. Spatial and channel-wise SE yields the overall highest performance, with an increase of Dice for brain segmentation and Dice for whole-body segmentation compared to the standard network. Particularly for brain, the performance increase is striking, given the limited increase in model complexity. Comparing the results across network architectures, DenseNets yield the best performance.
Fig. 2 and Fig. 3 present structure-wise results for whole brain and whole body segmentation, respectively, for DenseNets. In Fig. 2, we observe that sSE and scSE outperform the normal model consistently for all the structures. The cSE model outperforms the normal model in most structures except some challenging structures like 3rd/4th ventricles, amygdala and ventral DC where its performance degrades. One possible explanation could be the small size of these structures, which might have got overlooked by only exciting the channels. The performance of sSE and scSE is very close. For whole body segmentation, in Fig. 3, we observe a similar pattern.
Qualitative Results: Fig. 4 presents segmentation results for MRI T1 brain scan in Fig. 4(a-d) and for Whole body ceCT scans in Fig. 4(e-h). We show the input scan, ground truth annotations, DenseNet segmentation along with our proposed DenseNet+scSE segmentation. We highlight ROIs with a white box and red arrow, to show regions where inclusion of scSE block improved the segmentation. For MRI brain scan segmentation, we indicate the structure left putamen, which is under segmented using DenseNet (Fig. 4(c)), but the segmentation improves with the inclusion of the scSE block (Fig. 4(d)). For whole body ceCT, we indicate the spleen, which is over segmented using DenseNet (Fig. 4(g)), and which is rectified with adding scSE block (Fig. 4(h)).
We proposed the integration of squeeze & excitation blocks within F-CNNs for image segmentation. Further, we introduced the spatial squeeze & excitation, which outperforms the previously proposed channel-wise squeeze & excitation. We demonstrated that SE blocks yield a consistent improvement for three different F-CNN architectures and for two different segmentation applications. Hence, recalibration with SE blocks seems to be a fairly generic concept to boost performance in CNNs. Strikingly, the substantial increase in segmentation accuracy comes with a negligible increase in model complexity. With the seamless integration, we believe that squeeze & excitation can be a crucial component for neural networks in many medical applications.
-  Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. In NIPS pp. 1097-1105.
-  He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In CVPR, pp. 770-778, IEEE.
-  Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, Springer 2015, pp. 234-241.
-  Milletari F, Navab N, Ahmadi SA. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016, pp. 565-571, IEEE.
-  Roy, A.G., Conjeti, S., Sheet, D., Katouzian, A., Navab, N. and Wachinger, C., 2017, September. Error Corrective Boosting for Learning Fully Convolutional Networks with Limited Data. In MICCAI, pp. 231-239, Springer.
-  Long J, Shelhamer E, and Darrell T. Fully convolutional networks for semantic segmentation. In CVPR 2015, pp. 3431-40, IEEE.
-  Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. In ICCV 2015, pp. 1520-28, IEEE.
-  Badrinarayanan V, Kendall A, Cipolla R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. In arXiv preprint:1511.00561, 2015.
-  Jégou, S., Drozdzal, M., Vazquez, D., Romero, A. and Bengio, Y., 2017, July. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In CVPR Workshop, pp. 1175-1183, IEEE.
-  Hu, J., Shen, L. and Sun, G., 2017. Squeeze-and-excitation networks. ArXiv:1709.01507.
-  Landman, B, Warfield, S., Miccai Workshop on Multiatlas Labeling. In MICCAI Grand Challenge, 2012.
-  Marcus, D.S., Fotenos, A.F., Csernansky, J.G., Morris, J.C. and Buckner, R.L., 2010. Open access series of imaging studies: longitudinal MRI data in nondemented and demented older adults. Journal of cognitive neuroscience, 22(12), pp.2677-2684.
-  Jimenez-del-Toro, O., and others, 2016. Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: VISCERAL anatomy benchmarks. IEEE transactions on medical imaging, 35(11), pp.2459-2475.