A Joint Deep Learning Approach for Automated Liver and Tumor Segmentation

A Joint Deep Learning Approach for Automated Liver and Tumor Segmentation

Nadja Gruber Department of Mathematics, University of Innsbruck
Technikerstraße 13, 6020 Innsbruck, Austria
Stephan Antholzer Department of Mathematics, University of Innsbruck
Technikerstraße 13, 6020 Innsbruck, Austria
Werner Jaschke Department of Radiology
Medical University of Innsbruck
Anichststraße 35, 6020 Innsbruck
Christian Kremser Department of Radiology
Medical University of Innsbruck
Anichststraße 35, 6020 Innsbruck
Markus Haltmeier Department of Mathematics, University of Innsbruck
Technikerstraße 13, 6020 Innsbruck, Austria
February 20, 2019

Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer in adults, and the most common cause of death of people suffering from cirrhosis. The segmentation of liver lesions in CT images allows assessment of tumor load, treatment planning, prognosis and monitoring of treatment response. Manual segmentation is a very time-consuming task and in many cases, prone to inaccuracies and automatic tools for tumor detection and segmentation are desirable. In this paper, we use a network architecture that consists of two consecutive fully convolutional neural networks. The first network segments the liver whereas the second network segments the actual tumor inside the liver. Our network is trained on a subset of the LiTS (Liver Tumor Segmentation) challenge and evaluated on data provided from the radiological center in Innsbruck.

1 Introduction

Liver cancer remains associated with a high mortality rate, in part related to initial diagnosis at an advanced stage of disease. Prospects can be significantly improved by earlier treatment beginning, and analysis of CT images is a main diagnostic tool for early detection of liver tumors. Manual inspection and segmentation is a labor- and time-intensive process yielding relatively imprecise results in many cases. Thus, there is interest in developing automated strategies to aid in the early detection of lesions. Due to complex backgrounds, significant variations in location, shape and intensity across different patients, both, the automated liver segmentation and the further detection of tumors, remain challenging tasks.

Semantic segmentation of CT images has been an active area of research over the past few years. Recent developments of deep learning have dramatically improved the performance of artificial intelligence. Deep learning algorithms, especially deep convolutional neural networks (CNN) have considerably outperformed their competitors in medical imaging. One of the most successful CNNs architectures is the s-called U-Net [7], which has won several competitions in the field of biomedical image segmentation.

We investigate a deep learning strategy that jointly segments the liver and the lesions in CT images. As in [3], we use a network of formed of two consecutive U-nets. The first network performs liver segmentation, while the second one incorporates the output of the first network and segments the lesion. We propose a joint loss weighted loss function combining the outputs of both networks. The network is trained on a subset of the LiTS (Liver Tumor Segmentation) challenge and evaluated on different data collected at the radiological center in Innsbruck. For our initial experiments, we perform consecutive training, with which we already obtain quite accurate results.


Figure 1.1: Illustration of Network architecture for automated semantic liver and tumor segmentation. The model consists of two sequential U-Nets. The raw images are fed into the first network, and the output is a binary image. The original image multiplied by the obtained liver mask represents the input of the second U-Net. The final output is a binary image in which label is assigned to tumor.

2 Joint Deep Learning Approach

The overall architecture of our model is illustrated in Figure1.1. For the semantic segmentation task of liver and tumor segmentation, we propose a model consisting of two modified U-Nets. Related FCN architectures have been proposed in [8, 4, 2, 3].

2.1 Mathematical Modelling

In the following, let and denote the set of training images and the corresponding segmented images, respectively. Here the label stand for liver, for tumor and for background. For the task of semantic liver and tumor segmentation, we generate segmentation masks

representing binary images where class label 1 stands for the liver or tumor, and binary images where class label stands for tumor.

Our approach is to train two networks

that separately perform liver and tumor segmentation. In the fist step, the network is applied such that . After decision making by selecting a threshold , we obtain a liver mask that is applied to each input image. Additionally, we applied windowing pointwise to the intensity values, which results in new training data

These data serve is input and corresponding ground truth for training the second network . By selecting another thresholds, this gives a mask for the tumors.

The final classification can be performed in assigning a pixel to class label if , to class label if and , and class label 0 otherwise. The goal is to find the high dimensional parameter vectors and such that the overall classification error is small. This is achieved by minimizing a loss function that describes how well the network performs on the training data. Here we propose to use the joint loss function


where denotes the categorical cross-entropy-loss and the constant weights the importance of the two classification outcomes. It has been demonstrated in [1] that a joint loss function can improve results compared to sequential approaches for joint image reconstruction and segmentation.

2.2 Sequential U-Net Architecture

We implement our model using two U-Nets , , one on top of the other. The combined network architecture is shown in Figure 1.1. The inputs for both CNNs are grey-scale images of size and their outputs are binary images of size . While the input of the first U-Net is of the form displayed in Figure 3.1, the input of the second U-Nets is produced by the output of the first one as explained in Section 2.1.

In both networks, the input passes through an initial convolution layer and is then processed by a sequence of convolution blocks at decreasing resolutions (contracting path). The expanding path of the U-Net then reverses this downsampling process. Skip connections between down- and upsampling path intend to provide local information to the global information while upsampling. As final step the output of the network is passed to a linear classifier that outputs (via sigmoid) a probability for each pixel being within the liver/tumor. The model is implemented in Keras111https://keras.io/ with the TensorFlow backend222https://www.tensorflow.org/.

2.3 Optimization of the Model

For our initial results, we train the networks sequentially, which means that first we optimize for and then use the output of as input for . Specifically, for training the second U-net we minimize


Here ist the value of at pixel , and the indicator function declares whether belongs the the class tumor or not. The weight controls the relative importance assigned to the two classes.

Bothe models have been trained using stochastic gradient decent with momentum for 300 and 600 epochs, respectively. Each iteration takes about 70 seconds on NVIDIA standard GPU. To avoid overfitting, we applied a dropout of in the upsampling path. Both U-Nets were trained with a learning rate of 0.001 and categorical cross-entropy loss. Since the tumor area only accounts for a small area compared to the full size of the image, we applied balanced loss (2.2) in a second optimization of the network and reduced the learning rate to 0.0001. Comparison with the joint loss (2.1) is subject of future work.

3 Experimental Results

3.1 Datasets

The network training is run using a subset of the publicly available LiTS-challenge333https://competitions.codalab.org/competitions/17094 dataset containing variable kinds of liver lesions (HCC, metastasis, …). The dataset consists of CT scans coming from different clinical institutions. Trained radiologists have manually segmented annotation of the liver and tumors. All of the volumes were enhanced with a contrast agent, imaged in the portal venous phase. Each volume contains a variable number of axial slices with a resolution of pixels and an approximate slice thickness ranging from 0.7 to . The training is applied on 765 axial slices, 50 are used for validation and 50 for testing.

Further test data are provided by radiological center at the medical university of Innsbruck. The dataset contains CT scans of patients suffering from HCC and the belonging reference annotations were drafted by medical scientists. Because deep learning algorithms achieve better performance if the data has a consistent scale or distribution, all data are standardized to have intensity values between before starting the optimization.

Figure 3.1: Training data provided by LiTS challenge.

3.2 Evaluation on Test Data

Each pixel of the image is assigned to one of the two classes liver/other tissue and tumor/other tissue, respectively, with a certain probability. Results of the automated liver and tumor segmentation are visualized in Figure 3.2. Comparison with ground truth and segmented liver and tumor give rise to the assumption that our approach is highly promising for obtaining high performance metrics. To qualitatively evaluate performance, we applied some of the commonly used evaluation metrics in semantic image segmentation. Performance metrics are summarized in Table 1.

Figure 3.2: Results on HCC data. Top: liver segmentation results (red) compared to ground truth boundary (blue). Bottom: tumor segmentation result (red) compared to ground truth (blue).

3.2.1 AUC metric

Area under ROC Curve (AUC) is a performance metric for binary classification problems. We applied ROC analysis to find the threshold that achieves the best results for the tumor segmentation task. Due to the very low rate of false classified pixels (most of them has probability close to one or close to zero), we decided to restrict the ROC curve to pixels whose probability for belonging to class tumor lies between 0.01 and 0.99.

Figure 3.3: Histogram that displays the number of pixels predicted falling into class tumor with probability .

In Figure 3.4 we can see that the best restricted AUC value (rAUC) conducting is achieved by a very small value of . We further calculated the corresponding threshold and could achieve an improvement of the tumor segmentation results [6].

Figure 3.4: Restricted ROC curves for varying weights of the balanced loss. Setting the importance of the background pixels lower seems to considerably improve the tumor segmentation results.

3.2.2 Intersection over Union

For a more complete evaluation of the segmentation results we use class accuracy in conjunction with the so called IoU metric. The latter is essentially a method to quantify the percent overlap between the ground truth and the prediction output. The IoU measure gives the similarity between predicted and ground-truth regions for the object of interest. The formula for quantifying the IoU score is:


where TP, FP and FN denote the True Positive Rate, False Positive Rate and False Negative Rate, respectively.

Figure 3.5: Intersection over Union of tumor segmentation for balanced loss with . The light pink, light blue and pink areas mark the prediction mask, ground truth and Intersection over Union, respectively.

Since the segmentation task can be regarded as clustering of pixels, Rand index [5], which is a measure of the similarity between two data clusterings, has been proposed as a measure of segmentation performance. By and two segmentations are notated. In the following paragraph denotes the ground truth and the segmentation results obtained by the joint network. The function is defined as if pixels and are in same class and otherwise. One can see that that small differences in the location of object boundaries will increase the rand error slightly while merging or splitting of objects lead to a big increase of the Rand error.

We evaluate this model under usage of test and validation set from LiTS challenge data ( images). The evaluation metrics are summarized in Table 1. The liver segmentation evaluation scores indicate that our model performs remarkable good. Pixel accuracy, Intersection over union (IoU) and Rand Index (RI) have values very close to . IoU and Rand Index performance score of the tumor segmentation show that the application of balanced loss with , achieves the best results.

Data rAUC IoU RI
Liver 0.99985 0.93848 0.96278
Tumor 0.02 0.88 0.99959 0.77108 0.87056
0.05 0.77 0.99955 0.72609 0.84895
0.30 0.75 0.99949 0.73879 0.84329

Table 1: Performance Evaluation metrics for tumor segmentation model

4 Conclusions

The presented a deep learning framework for the automated joint liver and tumor segmentation. Segmentation metrics evaluate the segmentation of detected lesions and are comprised of a restricted AUC, an overlap Index (IoU) and Rand Index (RI). The current model works sequentially. A joint approach could be applied in order to reduce time required for computation and improve performances. Another interesting topic to address is the classification of the tumors detected by a deep learning algorithm.


  • [1] Jonas Adler, Sebastian Lunz, Olivier Verdier, Carola-Bibiane Schönlieb, and Ozan Öktem. Task adapted reconstruction for inverse problems. arXiv:1809.00948, 2018.
  • [2] Grzegorz Chlebus, Andrea Schenk, Jan Hendrik Moltz, Bram van Ginneken, Horst Karl Hahn, and Hans Meine. Deep learning based automatic liver tumor segmentation in ct with shape-based post-processing. 2018.
  • [3] Patrick Ferdinand Christ, Florian Ettlinger, et al. Automatic liver and tumor segmentation of ct and mri volumes using cascaded fully convolutional neural networks. arXiv:1702.05970, 2017.
  • [4] Xiao Han. Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv:1704.07239, 2017.
  • [5] Viren Jain, Benjamin Bollmann, Mark Richardson, Daniel R Berger, Moritz N Helmstaedter, Kevin L Briggman, Winfried Denk, Jared B Bowden, John M Mendenhall, Wickliffe C Abraham, et al. Boundary learning by optimization with topological constraints. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2488–2495. IEEE, 2010.
  • [6] Rajeev Kumar and Abhaya Indrayan. Receiver operating characteristic (roc) curve for medical researchers. Indian pediatrics, 48(4):277–287, 2011.
  • [7] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  • [8] Eugene Vorontsov, An Tang, Chris Pal, and Samuel Kadoury. Liver lesion segmentation informed by joint liver segmentation. In Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on, pages 1332–1335. IEEE, 2018.
Comments 2
Request Comment
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description