2D-Densely Connected Convolution Neural Networks for automatic Liver and Tumor Segmentation
In this paper we propose a fully automatic 2-stage cascaded approach for segmentation of liver and its tumors in CT (Computed Tomography) images using densely connected fully convolutional neural network (DenseNet). We independently train liver and tumor segmentation models and cascade them for a combined segmentation of the liver and its tumor. The first stage involves segmentation of liver and the second stage uses the first stage’s segmentation results for localization of liver and henceforth tumor segmentations inside liver region. The liver model was trained on the down-sampled axial slices , whereas for the tumor model no down-sampling of slices was done, but instead it was trained on the CT axial slices windowed at three different Hounsfield (HU) levels. On the test set our model achieved a global dice score of 0.923 and 0.625 on liver and tumor respectively. The computed tumor burden had an rmse of 0.044.
I Introduction and Related Work
The exponential growth in the number of medical images of different modalities used in clinical practice has led to an interest in developing automated methods for analysis of medical images. Computed tomography (CT scan) is a noninvasive diagnostic imaging procedure that uses a combination of X-rays and computer algorithms to generate different internal views of the body (often called slices). A CT scan shows detailed images of internal organs and structures inside the body, which also includes the bones, muscles and fat. CT scans of the abdomen and pelvis is used as a diagnostic imaging test to aid in the detection of diseases of liver, biliary tract, the small intestine and colon. CT scanning is painless, noninvasive and accurate. When compared to Magnetic Resonance Imaging (MRI), CT has wider availability, and is fast. CT images are generated using X-ray beams and the amount of X-rays absorbed by tissues at each location in the body is mapped to Hounsfield units (HU). The denser the tissue, the more the X-rays are attenuated, and the higher the number of HU. Water is always set to be 0 HU, while air is â1000 HU, and bones have values between several hundred to several thousand HU.
Manual segmentation of the organs and tumors from medical images is a tedious task and often introduces inter-rater variability. Convolutional neural networks (CNNs)  have been applied to wide variety of image classification [2, 3, 4] and semantic segmentation [5, 6] tasks. In this paper, we focus on segmentation of liver and its tumors in CT images taken from thoracic to pelvis region using CNNs. Our network’s architecture for segmentation task is inspired from DenseNet [10, 9]. DenseNet connects each layer to every other layer in a feed-forward fashion by concatenation of all feature outputs. The output of the layer is defined as
where represents the feature maps at the layer and represents the concatenation operation. In our case, is the layer comprising of Batch Normalization (BN) , followed by Exponential Linear Unit (ELU) , a convolution and dropout . This kind of connectivity pattern aids in reuse of features and allows implicit deep supervision during training and thus substantially reduces the number of parameters while maintaining good performance, which is ideal in scenarios with limited data. The output dimension of each layer has (growth rate parameter) feature maps. The number of feature maps in DenseNet grow linearly with depth. A Transition Down (TD) layer in DenseNet is introduced for reducing spatial dimension of feature maps which is accomplished by using a convolution (depth preserving) followed by a max-pooling operation. A denseblock refers to concatenation of new feature maps created at a given resolution.
Ii Our Method
CT Windowing is a technique frequently used in the evaluation of CT scans for the purpose of enhancing contrast of particular type of tissue or the abnormality type being examined. The abdominal CT images of a patient comprises of various organs such as liver, spleen, gal bladder, etc. Anatomically, the HU range of liver is . Our method is based on 2-stage cascaded approach  for segmentation of liver and its tumors from HU windowed CT volumes. In this method we first train the liver model for the task of liver segmentation. Since the shape (contour) and texture of liver is simple when compared to its tumors, the CT images after windowing were down-sampled to half their original size () and then used for training. This helped in reducing the computation required for the training liver model and also helps in faster segmentation of liver. The tumor model was 3-channel input and was trained independently on full-sized CT images of liver windowed at different levels having different window widths. During prediction, the liver segmentation from first stage in the cascade (liver model) aids the second stage (tumor model) by precisely localizing on liver regions in CT images to produce combined predictions of liver and tumor segmentations.
Ii-a Dataset and Preprocessing
The models were trained and tested on the LITS MICCAI-2017 challenge dataset which comprised of contrast enhanced CT images taken at different phases (mostly venous phase) with only a few cases with anomalies like fatty liver, cirrhosis liver, calcification in the liver, etc. Out of CT scans, for the scans radiologist hand-drawn ground-truths were given for training the model and the rest were used for testing by the challenge organizers. We divided the training dataset into train:â volumes, validation:â volumes, test:â volumes.
For liver model, the following pre-processing techniques were done on the CT volumes in the order specified:
HU values are windowed to the range of [-100,300].
min-max normalization on the entire volume.
Down-sample the slices from to .
For tumor model, the following pre-processing techniques were done on the CT volumes in the order specified:
3 different HU windowing ranges ([0,100], [-100,200], [-100,400]) were used to produce 3 images.
min-max normalization on the entire volume.
In most of the CT volumes (in the challenge dataset), the liver and tumor slices comprised of a small fraction of the total volume. So in order to address the data imbalance, the liver model was trained only on liver slices and additional 10 slices were taken above and below the liver. Similarly, the tumor model was trained only on tumor slices and with additional 5 slices above and below the tumor.
Ii-B Liver Model
Ii-B1 Network Architecture
Fig.(1) illustrates the schematic diagram of the proposed network for liver segmentation. Our proposed network doesnât have skip connections. The up-sampling and down-sampling paths uses fully convolutional denseblocks. Each denseblock comprises of 4 layer blocks and each layer in the denseblock is sequentially composed of BN ELU convolution layer. The first Dense-Block was prefixed with a layer comprising of several convolution filters of size on the input images. In the down-sampling path, the input to a dense block was concatenated with its output, leading to a linear growth of the number of feature maps. The Transition-Down block (TD) consists of BN ELU convolution and max-pooling layers. The last layer of the down-sampling path is referred to as Bottleneck.
In the up-sampling path, the input of a Dense-Block is not concatenated with its output. Transition-Up (TU) block consists of spatial-bilinear up-sampler followed by BN ELU and convolution layer. During our training we found that by using spatial bilinear up-sampler followed by convolution yielded better results compared to transposed convolution (deconvolution) operation with a stride of . Our thesis is that the network also learns better way of up-sampling the outputs which resulted in better predictions. To get the final segmentation map of size of a simple spatial bilinear up-sampling block (SBU) is added in the penultimate layer. The feature maps of the hindmost up-sampling component was convolved with a convolution layer followed by a soft-max layer to generate the final segmentation. To prevent over-fitting, a dropout rate of was implemented following each convolution layer.
Table (IV) summaries the individual blocks of network architecture.
It was observed that by using Exponential Linear Unit (ELUs) instead of Rectified Linear units (ReLUs) led to faster convergence.
Ii-B2 Loss function
Liver being the largest organ in the body, the class imbalance shown in the CT volume is minimal, but contouring of the liver borders from its neighboring organs in CT images is generally not precise. Hence in-order to predict the edges precisely, weight-maps were generated from the given ground truth segmentation (see Figure 2), edges of the liver were weighed higher relative to the interior regions of the liver. These weight-maps were used during training for minimizing the spatially-weighted-cross-entropy loss function. The use of weight-mapping leads to adding a heavy penalty in cost function for predicting imprecise liver contours during training the model.
In our experiments we define an epoch to be completed when a specified number of iterations are executed on the randomly sampled batches of data from the train and validation set. In this experiment the number of iterations were and for train and validation datasets respectively.
Ii-B3 Post processing
The outputs of the Liver network were subjected to the following post-processing methods sequentially:
Morphological binary erosion
Get the largest connected component
Morphological binary dilation
By applying these techniques, we were able to remove some false positives like spleen or heart that might be very closely attached to the liver.
Ii-C Tumor Model
Ii-C1 Network Architecture
The network architecture for tumor model was similar to liver model, except that it accepted 3-channel inputs and there was no simple spatial bilinear up-sampling block (SBU) as in the liver model. The 3 channels were feed with the same slice of the CT volume, but each channelâs HU - windowing was at 3 different levels having different window widths. It was observed that by providing 3 different HU windowed channels, the model is able to learn the tumor and its boundary very well compared to a single channel input. To prevent over-fitting a dropout rate of was implemented.
Ii-C2 Loss function
The shape of tumors is diffusive, inhomogeneous and sparsely located in the CT volumes. This leads to class imbalance in the dataset, thereby making it hard for the network to learn the intricate features of the tumor. Hence in order to reduce the class imbalance we employ two strategies:
Use of weight-maps with tumor portion being weighed very high compared to background. The weight-maps are used for calculation of spatially-weighted-cross-entropy loss.
Use weighted combination of two loss functions: spatially-weighted-cross-entropy loss and a loss function based on dice overlap co-efficient.
The dice co-efficient is an overlap metric used for assessing the quality of segmentation maps. The dice coefficient between two binary volumes can be written as:
where the sums run over the voxels, of the predicted binary segmentation volume and the ground truth binary volume
The parameters of the network were optimized so as to minimize the , Equation. (4).
where and are empirically assigned weights to individual losses. During training it was observed that the dice loss allowed higher overlap scores than when trained with the loss function based on the cross entropy loss alone. In this work we set and .
Ii-C3 Post processing
The predictions of the tumor network were masked with the background taken from the liver prediction, hence the final tumor predictions were only inside the liver.
With the pre-processing, network architecture and post processing steps discussed in this paper we were able to achieve an average liver dice of 0.93 on the 14 test volumes (mentioned in the dataset split up in the beginning) and an average tumor dice of 0.45 on the 14 test volumes. Our liver predictions have a smoother surface (3D volume) and precise edges compared to the liver predictions from a U-net whose predictions have ridges on the surface that arise because of very finer slice wise predictions that result because of the skip connections. The true positives of tumor prediction have almost more than 50% overlap with the ground truth but the model predicts a lot of false positives like gallbladder edges, diaphragm, vessels etc. The results of our proposed 2-stage cascaded model’s predictions on the 70 CT test volumes is summarized in Tabels V & VI.
|Computed LESION DETECTION metrics||Computed TUMOR BURDEN|
|recall: 0.348||rmse: 0.044|
|precision_greater_zero: 0.211||max: 0.194|
Appendix A Segmentation Results
-  LeCun, Yann, et al. ”Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324.
-  Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. ”Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
-  Simonyan, Karen, and Andrew Zisserman. ”Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).
-  He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
-  Long, Jonathan, Evan Shelhamer, and Trevor Darrell. ”Fully convolutional networks for semantic segmentation.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.
-  Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” arXiv preprint arXiv:1505.04597 (2015).
-  Ciresan, Dan, et al. ”Deep neural networks segment neuronal membranes in electron microscopy images.” Advances in neural information processing systems. 2012.
-  Menze, Bjoern H., et al. ”The multimodal brain tumor image segmentation benchmark (BRATS).” IEEE transactions on medical imaging 34.10 (2015): 1993-2024.
-  JÃ©gou, Simon, et al. ”The one hundred layers tiramisu: Fully convolutional DenseNets for semantic segmentation.” arXiv preprint arXiv:1611.09326 (2016).
-  Huang, Gao, et al. ”Densely connected convolutional networks.” arXiv preprint arXiv:1608.06993 (2016).
-  Christ, Patrick Ferdinand, et al. ”Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2016.
-  MartÃn Abadi, et al. ”TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org”.
-  StÃ©fan van der Walt, Johannes L. SchÃ¶nberger, Juan Nunez-Iglesias, FranÃ§ois Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu and the scikit-image contributors. ”scikit-image: Image processing in Python. PeerJ 2:e453 (2014)”
-  Kingma, Diederik, and Jimmy Ba. ”Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).
-  Srivastava, Nitish, et al. ”Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research 15.1 (2014): 1929-1958.
-  Ioffe, Sergey, and Christian Szegedy. ”Batch normalization: Accelerating deep network training by reducing internal covariate shift.” International Conference on Machine Learning. 2015.
-  Clevert, Djork-ArnÃ©, Thomas Unterthiner, and Sepp Hochreiter. ”Fast and accurate deep network learning by exponential linear units (elus).” arXiv preprint arXiv:1511.07289 (2015).