2D-Densely Connected Convolution Neural Networks for automatic Liver and Tumor Segmentation

2D-Densely Connected Convolution Neural Networks for automatic Liver and Tumor Segmentation

Krishna Chaitanya Kaluva, Mahendra Khened, Avinash Kori and Ganapathy Krishnamurthi KC. Kaluva, M. Khened, A. Kori and G. Krishnamurthi are with the Department of Engineering Design, Indian Institute of Technology, Madras, Chennai, 600036 India email: (gankrish@iitm.ac.in).Manuscript received August 13, 2017; revised August 15, 2017.

In this paper we propose a fully automatic 2-stage cascaded approach for segmentation of liver and its tumors in CT (Computed Tomography) images using densely connected fully convolutional neural network (DenseNet). We independently train liver and tumor segmentation models and cascade them for a combined segmentation of the liver and its tumor. The first stage involves segmentation of liver and the second stage uses the first stage’s segmentation results for localization of liver and henceforth tumor segmentations inside liver region. The liver model was trained on the down-sampled axial slices , whereas for the tumor model no down-sampling of slices was done, but instead it was trained on the CT axial slices windowed at three different Hounsfield (HU) levels. On the test set our model achieved a global dice score of 0.923 and 0.625 on liver and tumor respectively. The computed tumor burden had an rmse of 0.044.

CNN, FCN, DenseNet, Liver Tumor, dice loss.


I Introduction and Related Work

The exponential growth in the number of medical images of different modalities used in clinical practice has led to an interest in developing automated methods for analysis of medical images. Computed tomography (CT scan) is a noninvasive diagnostic imaging procedure that uses a combination of X-rays and computer algorithms to generate different internal views of the body (often called slices). A CT scan shows detailed images of internal organs and structures inside the body, which also includes the bones, muscles and fat. CT scans of the abdomen and pelvis is used as a diagnostic imaging test to aid in the detection of diseases of liver, biliary tract, the small intestine and colon. CT scanning is painless, noninvasive and accurate. When compared to Magnetic Resonance Imaging (MRI), CT has wider availability, and is fast. CT images are generated using X-ray beams and the amount of X-rays absorbed by tissues at each location in the body is mapped to Hounsfield units (HU). The denser the tissue, the more the X-rays are attenuated, and the higher the number of HU. Water is always set to be 0 HU, while air is −1000 HU, and bones have values between several hundred to several thousand HU.

Manual segmentation of the organs and tumors from medical images is a tedious task and often introduces inter-rater variability. Convolutional neural networks (CNNs) [1] have been applied to wide variety of image classification [2, 3, 4] and semantic segmentation [5, 6] tasks. In this paper, we focus on segmentation of liver and its tumors in CT images taken from thoracic to pelvis region using CNNs. Our network’s architecture for segmentation task is inspired from DenseNet [10, 9]. DenseNet connects each layer to every other layer in a feed-forward fashion by concatenation of all feature outputs. The output of the layer is defined as


where represents the feature maps at the layer and represents the concatenation operation. In our case, is the layer comprising of Batch Normalization (BN) [16], followed by Exponential Linear Unit (ELU) [17], a convolution and dropout [15]. This kind of connectivity pattern aids in reuse of features and allows implicit deep supervision during training and thus substantially reduces the number of parameters while maintaining good performance, which is ideal in scenarios with limited data. The output dimension of each layer has (growth rate parameter) feature maps. The number of feature maps in DenseNet grow linearly with depth. A Transition Down (TD) layer in DenseNet is introduced for reducing spatial dimension of feature maps which is accomplished by using a convolution (depth preserving) followed by a max-pooling operation. A denseblock refers to concatenation of new feature maps created at a given resolution.

Ii Our Method

CT Windowing is a technique frequently used in the evaluation of CT scans for the purpose of enhancing contrast of particular type of tissue or the abnormality type being examined. The abdominal CT images of a patient comprises of various organs such as liver, spleen, gal bladder, etc. Anatomically, the HU range of liver is . Our method is based on 2-stage cascaded approach [11] for segmentation of liver and its tumors from HU windowed CT volumes. In this method we first train the liver model for the task of liver segmentation. Since the shape (contour) and texture of liver is simple when compared to its tumors, the CT images after windowing were down-sampled to half their original size () and then used for training. This helped in reducing the computation required for the training liver model and also helps in faster segmentation of liver. The tumor model was 3-channel input and was trained independently on full-sized CT images of liver windowed at different levels having different window widths. During prediction, the liver segmentation from first stage in the cascade (liver model) aids the second stage (tumor model) by precisely localizing on liver regions in CT images to produce combined predictions of liver and tumor segmentations.

Ii-a Dataset and Preprocessing

The models were trained and tested on the LITS MICCAI-2017 challenge dataset which comprised of contrast enhanced CT images taken at different phases (mostly venous phase) with only a few cases with anomalies like fatty liver, cirrhosis liver, calcification in the liver, etc. Out of CT scans, for the scans radiologist hand-drawn ground-truths were given for training the model and the rest were used for testing by the challenge organizers. We divided the training dataset into train:– volumes, validation:– volumes, test:– volumes.

For liver model, the following pre-processing techniques were done on the CT volumes in the order specified:

  1. HU values are windowed to the range of [-100,300].

  2. min-max normalization on the entire volume.

  3. Down-sample the slices from to .

For tumor model, the following pre-processing techniques were done on the CT volumes in the order specified:

  1. 3 different HU windowing ranges ([0,100], [-100,200], [-100,400]) were used to produce 3 images.

  2. min-max normalization on the entire volume.

In most of the CT volumes (in the challenge dataset), the liver and tumor slices comprised of a small fraction of the total volume. So in order to address the data imbalance, the liver model was trained only on liver slices and additional 10 slices were taken above and below the liver. Similarly, the tumor model was trained only on tumor slices and with additional 5 slices above and below the tumor.

Ii-B Liver Model

Ii-B1 Network Architecture

Fig. 1: Network Architecture of liver model.

Fig.(1) illustrates the schematic diagram of the proposed network for liver segmentation. Our proposed network doesn’t have skip connections. The up-sampling and down-sampling paths uses fully convolutional denseblocks. Each denseblock comprises of 4 layer blocks and each layer in the denseblock is sequentially composed of BN ELU convolution layer. The first Dense-Block was prefixed with a layer comprising of several convolution filters of size on the input images. In the down-sampling path, the input to a dense block was concatenated with its output, leading to a linear growth of the number of feature maps. The Transition-Down block (TD) consists of BN ELU convolution and max-pooling layers. The last layer of the down-sampling path is referred to as Bottleneck.
In the up-sampling path, the input of a Dense-Block is not concatenated with its output. Transition-Up (TU) block consists of spatial-bilinear up-sampler followed by BN ELU and convolution layer. During our training we found that by using spatial bilinear up-sampler followed by convolution yielded better results compared to transposed convolution (deconvolution) operation with a stride of . Our thesis is that the network also learns better way of up-sampling the outputs which resulted in better predictions. To get the final segmentation map of size of a simple spatial bilinear up-sampling block (SBU) is added in the penultimate layer. The feature maps of the hindmost up-sampling component was convolved with a convolution layer followed by a soft-max layer to generate the final segmentation. To prevent over-fitting, a dropout rate of was implemented following each convolution layer.


Layer Batch Normalization Exponential Linear Unit Convolution Dropout


TD Batch Normalization Exponential Linear Unit Convolution Dropout Max Pooling


TU Spatial Bilinear Up-sampler Convolution Batch Normalization

TABLE IV: Building blocks of the network architecture. From left to right: layer used in the model, Transition Down (TD) and Transition Up (TU).

Table (IV) summaries the individual blocks of network architecture.

It was observed that by using Exponential Linear Unit (ELUs) instead of Rectified Linear units (ReLUs) led to faster convergence.

Ii-B2 Loss function

Liver being the largest organ in the body, the class imbalance shown in the CT volume is minimal, but contouring of the liver borders from its neighboring organs in CT images is generally not precise. Hence in-order to predict the edges precisely, weight-maps were generated from the given ground truth segmentation (see Figure 2), edges of the liver were weighed higher relative to the interior regions of the liver. These weight-maps were used during training for minimizing the spatially-weighted-cross-entropy loss function. The use of weight-mapping leads to adding a heavy penalty in cost function for predicting imprecise liver contours during training the model.

(a) Ground Truth Segmentation
(b) Weight Map
Fig. 2: The figure shows the weight-map generation from the ground-truth label image

In our experiments we define an epoch to be completed when a specified number of iterations are executed on the randomly sampled batches of data from the train and validation set. In this experiment the number of iterations were and for train and validation datasets respectively.

The parameters of the network were optimized so as to minimize the , Equation. (2). The liver model was trained with a batch size of for epochs with a learning rate of and L2 weight-decay of using ADAM optimizer[14].

Ii-B3 Post processing

The outputs of the Liver network were subjected to the following post-processing methods sequentially:

  1. Morphological binary erosion

  2. Get the largest connected component

  3. Morphological binary dilation

By applying these techniques, we were able to remove some false positives like spleen or heart that might be very closely attached to the liver.

Ii-C Tumor Model

Ii-C1 Network Architecture

The network architecture for tumor model was similar to liver model, except that it accepted 3-channel inputs and there was no simple spatial bilinear up-sampling block (SBU) as in the liver model. The 3 channels were feed with the same slice of the CT volume, but each channel’s HU - windowing was at 3 different levels having different window widths. It was observed that by providing 3 different HU windowed channels, the model is able to learn the tumor and its boundary very well compared to a single channel input. To prevent over-fitting a dropout rate of was implemented.

Ii-C2 Loss function

The shape of tumors is diffusive, inhomogeneous and sparsely located in the CT volumes. This leads to class imbalance in the dataset, thereby making it hard for the network to learn the intricate features of the tumor. Hence in order to reduce the class imbalance we employ two strategies:

  • Use of weight-maps with tumor portion being weighed very high compared to background. The weight-maps are used for calculation of spatially-weighted-cross-entropy loss.

  • Use weighted combination of two loss functions: spatially-weighted-cross-entropy loss and a loss function based on dice overlap co-efficient.

The dice co-efficient is an overlap metric used for assessing the quality of segmentation maps. The dice coefficient between two binary volumes can be written as:


where the sums run over the voxels, of the predicted binary segmentation volume and the ground truth binary volume

The parameters of the network were optimized so as to minimize the , Equation. (4).


where and are empirically assigned weights to individual losses. During training it was observed that the dice loss allowed higher overlap scores than when trained with the loss function based on the cross entropy loss alone. In this work we set and .

Ii-C3 Post processing

The predictions of the tumor network were masked with the background taken from the liver prediction, hence the final tumor predictions were only inside the liver.

Iii Conclusion

With the pre-processing, network architecture and post processing steps discussed in this paper we were able to achieve an average liver dice of 0.93 on the 14 test volumes (mentioned in the dataset split up in the beginning) and an average tumor dice of 0.45 on the 14 test volumes. Our liver predictions have a smoother surface (3D volume) and precise edges compared to the liver predictions from a U-net whose predictions have ridges on the surface that arise because of very finer slice wise predictions that result because of the skip connections. The true positives of tumor prediction have almost more than 50% overlap with the ground truth but the model predicts a lot of false positives like gallbladder edges, diaphragm, vessels etc. The results of our proposed 2-stage cascaded model’s predictions on the 70 CT test volumes is summarized in Tabels V & VI.

Computed LIVER
Computed LESION
voe 0.150 0.411
dice global 0.923 0.625
dice 0.912 0.725
rmsd 9.682 2.070
rvd -0.008 19.705
assd 6.465 1.441
jaccard 0.850 0.589
dice_per_case 0.912 0.492
msd 45.928 7.515
TABLE V: Results of segmentation metrics on the test set comprising of 70 CT volumes
recall: 0.348 rmse: 0.044
precision_greater_zero: 0.211 max: 0.194
precision: 0.117
recall_greater_zero: 0.628
TABLE VI: Results of lesion detection and tumor burden estimation metrics on the test set comprising of 70 CT volumes

Appendix A Segmentation Results

Fig. 3: The figure shows the segmentation results of liver and tumor in a CT slice of test dataset. The color red indicates liver and green indicates tumor. The proposed model sometimes mis classifies darker vessels as tumors.


  • [1] LeCun, Yann, et al. ”Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324.
  • [2] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. ”Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
  • [3] Simonyan, Karen, and Andrew Zisserman. ”Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).
  • [4] He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  • [5] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. ”Fully convolutional networks for semantic segmentation.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.
  • [6] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” arXiv preprint arXiv:1505.04597 (2015).
  • [7] Ciresan, Dan, et al. ”Deep neural networks segment neuronal membranes in electron microscopy images.” Advances in neural information processing systems. 2012.
  • [8] Menze, Bjoern H., et al. ”The multimodal brain tumor image segmentation benchmark (BRATS).” IEEE transactions on medical imaging 34.10 (2015): 1993-2024.
  • [9] Jégou, Simon, et al. ”The one hundred layers tiramisu: Fully convolutional DenseNets for semantic segmentation.” arXiv preprint arXiv:1611.09326 (2016).
  • [10] Huang, Gao, et al. ”Densely connected convolutional networks.” arXiv preprint arXiv:1608.06993 (2016).
  • [11] Christ, Patrick Ferdinand, et al. ”Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2016.
  • [12] Martín Abadi, et al. ”TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org”.
  • [13] Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu and the scikit-image contributors. ”scikit-image: Image processing in Python. PeerJ 2:e453 (2014)”
  • [14] Kingma, Diederik, and Jimmy Ba. ”Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).
  • [15] Srivastava, Nitish, et al. ”Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research 15.1 (2014): 1929-1958.
  • [16] Ioffe, Sergey, and Christian Szegedy. ”Batch normalization: Accelerating deep network training by reducing internal covariate shift.” International Conference on Machine Learning. 2015.
  • [17] Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. ”Fast and accurate deep network learning by exponential linear units (elus).” arXiv preprint arXiv:1511.07289 (2015).
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description