A survey on shape-constraint deep learning for medical image segmentation

A survey on shape-constraint deep learning for medical image segmentation

Abstract

Since the advent of U-Net, fully convolutional deep neural networks and its many variants have completely changed the modern landscape of deep learning based medical image segmentation. However, the over dependence of these methods on pixel level classification and regression has been identified early on as a problem. Especially when trained on medical databases with sparse available annotation, these methods are prone to generate segmentation artifacts such as fragmented structures, topological inconsistencies and islands of pixel. These artefacts are especially problematic in medical imaging since segmentation is almost always a pre-processing step for some downstream evaluation. The range of possible downstream evaluations is rather big, for example surgical planning, visualization, shape analysis, prognosis, treatment planning etc. However, one common thread across all these downstream tasks is the demand of anatomical consistency. To ensure the segmentation result is anatomically consistent, approaches based on Markov/ Conditional Random Fields, Statistical Shape Models are becoming increasingly popular over the past 5 years. In this review paper, a broad overview of recent literature on bringing anatomical constraints for medical image segmentation is given, the shortcomings and opportunities of the proposed methods are thoroughly discussed and potential future work is elaborated. We review the most relevant papers published until the submission date. For quick access, important details such as the underlying method, datasets and performance are tabulated.

\keywords

Medical Image Segmentation Shape Priors Shape Models CRF MRF Active Contours

1 Introduction

Semantic segmentation is the task of predicting the category of individual pixels in the image which has been one of the key problems in the field of image understanding and computer vision for a long time. It has a vast range of applications such as autonomous driving (detecting road signs, pedestrians and other road users), land use and land cover classification, image search engines, medical field (detecting and localizing the surgical instruments, describing the brain tumors, identifying organs in different image modalities). This problem has been tackled by a combination of machine learning and computer vision, approaches in the past. Despite their popularity and success, deep learning era changed main trends. Many of the problems in computer vision - semantic segmentation among them - have been solved with convolutional neural networks (CNNs) .

Incorporating prior knowledge into traditional image segmentation algorithms has proven useful for obtaining more accurate and plausible results. The highly constrained nature of anatomical objects can be well captured with learning based techniques. However, in most recent and promising techniques such as CNN based segmentation it is not obvious how to incorporate such prior knowledge. Segmenting images that suffer from low-quality and low signal-to-noise ratio without any shape constraint remains problematic even for CNNs. Though it has been shown that incorporation of shape prior information significantly improves the performance of the segmentation algorithms, incorporation of such prior knowledge is a tricky practical problem. In this work, we provide an overview of efforts of shape prior usage in deep learning frameworks.

1.1 Yet another review paper

There already appeared a variety of review papers about shape modelling and deep learning for medical image segmentation in the recent past. McInerney and Terzopoulos (1996) presents various approaches that apply deformable models. Peng et al. (2013) deals with different categories of graph-based models where meaningful objects are represented by sub-graphs. The review by Heimann and Meinzer (2009) is about statistical shape models and concentrates especially on landmark-based shape representations. Elnakib et al. (2011) also reviews different shape feature based models, that include statistical shape models, as well as deformable models. A more recent review by Nosrati and Hamarneh (2016) provides insights into segmentation models that incorporate shape information as prior knowledge. Later surveys of Litjens et al. (2017), Razzak et al. (2017), Rizwan I Haque and Neubert (2020) and Lei et al. (2020) shift their focus to deep learning approaches. Hesamian et al. (2019) and Taghanaki et al. (2019) present different network architectures and training techniques, whereas Jurdi et al. (2020) take it a step further and reviews prior-based loss functions in neural networks.

Since deep learning became the method of choice for many computer vision tasks, including medical image segmentation, we focus our review on models that combine neural networks with explicit shape models in order to incorporate shape knowledge into the segmentation process. Segmentation models solely based on neural networks usually do not incorporate any form of shape knowledge. They are based on traditional loss functions that only regard objects at pixel level and do not evaluate global structures. The papers we present in this review improve these networks by combining them with additional models that are especially built with shape in mind. This is also the point that delimits this review from existing surveys which either focus mostly deep learning approaches or on traditional shape and deformable model methods, but not on the combination of both.

The explicit models applied in this review can be divided into three main categories as shown in Figure 1: 1) Conditional or Markov Field models that establish connections between different pixel regions 2) Active/Statistical Shape Models that learn a special representation for valid shapes 3) Active Contour Models or snakes that use deformable splines for shape detection. These models are either applied as pre-processing steps to create initial segmentations, post-processing steps to refine the neural network segmentations, or used in multi-step models consisting of various models along a specific pipeline.

We are aware that the field is heavily shifting from explicit ways of modeling shape to more implicit approaches where networks are trained in an end-to-end way.Up and coming Works propose more intelligent loss functions that no longer require additional explicit shape modelling, but only consist of a single neural network. Zhang et al. (2020a) proposed a new geometric loss for lesion segmentation. Other examples are Mohagheghi and Foruzan (2020) and Han et al. (2020) where the loss contains shape priors. Li et al. (2020) introduces a spatially encoded loss with a special shape attention mechanism. Clough et al. (2019b) uses a topology based loss function.

However the overwhelming majority of articles combine neural networks and explicit models to introduce shape knowledge. This combination often stems from a rather principled engineering design choice (as shown in Figure 1) which is not detailed in any of the previous review articles. This review focuses on this overarching design principle of shape constraint which, along with being a quick access guide to explicit approaches, will work as a research catalyzer of implicit constraints.

Figure 1: Overview of related work approaches

2 CRF / MRF approaches

Markov Random Fields (MRF) Li (1994) belong to the domain of graphical models and model relationships between pixels or high-level features with a neighborhood system. The label probability of a single pixel is thereby conditioned on all neighboring pixels which allow to model contextual constraints. The maximum a posteriori probability (MAP) can then be calculated by applying the Bayes rule. Conditional Random Fields (CRF) Lafferty et al. (2001) are an extension of MRFs and allow to incorporate arbitrary global features over regions of pixels. For medical image segmentation this means that they generate smooth edges by using this global knowledge about surrounding regions which is a reason why the are often applied alongside neural networks to perform medical image segmentation.

CRFs used for postprocessing

The largest category of methods that utilize CRFs or MRFs apply them as a pos-tprocessing step. A large portion of papers focus on the straight-forward approach where the CNN generates initial segmentations maps which are directly passed to a CRF or MRF model as inputs for further refinements. These approaches are evaluated on a variety of anatomies and mostly differ in the utilized network architectures but follow the same idea. They are applied to lung nodules (Yaguchi et al. (2019), Gao et al. (2016)), retinal vessel (Fu et al. (2016b)), brain tumor (Zhao et al. (2016), Li et al. (2017)), cervical nuclei (Liu et al. (2018)), eye sclera (Mesbah et al. (2017)), melanoma (Luo and Yang (2018)), ocular structure (Nguyen et al. (2018)), left atrial appendage (Jin et al. (2018)), lymph node (Nogues et al. (2016)), liver (Dou et al. (2016)) and prostate cancer lesion (Cao et al. (2019)) segmentation tasks. A slightly different approach for skin lesion detection by Qiu et al. (2020) is based on the same idea, but uses not just a single CNN network, but an ensemble of seven or fifteen which are combined inside the CRF. Two other approaches to highlight here for brain region (Zhai and Li (2019)) and optical discs in fundus image (Bhatkalkar et al. (2020)) segmentation integrate a special attention mechanism into their networks with the motivation to improve the segmentations by detecting and exploiting salient deep features. Another special version that operates on weakly segmented bounding box images for fetal brain & lung segmentation is introduced by Rajchl et al. (2017). Given the initial weak segmentations, the model iteratively optimizes the pixel predictions with a CNN followed by a CRF to obtain the final segmentation maps.
Instead of CRFs, Shakeri et al. (2016) use a MRF to impose volumetric homogenity on the outputs of a CNN for subcortical region segmentation. MRFs are also utilized in the approach shown by Xia et al. (2019) for kidney segmentation where the MRF is integrated into a SIFT-Flow model.
Besides these classical approaches, another method that came up focused on cascading CNN networks that generate segmentations in a coarse-to-fine fashion. Wachinger et al. (2018) use this strategy with a first network that segments fore- from background pixels in brain MRIs and a second one that classifies the actual brain regions. The same method is also used by Shen and Zhang (2017) for brain tumor segmentation, by Dou et al. (2017) for liver and whole heart segmentation, and by Christ et al. (2016) for liver-based lesion segmentation.
A somewhat different cascading structure, for brain tumor segmentation, is introduced by Hu et al. (2019) where multiple subsequent CNNs are used to extract more discriminative multi-scale features and to capture dependencies. Feng et al. (2020) extend this version on the task of brain tumor segmentation with the introduction of residual connections that improve the overall performance. Similar to the cascading methods, there are CNNs with two pathways that combine two parallel networks on different resolution levels that aim for capturing larger 3D contexts. The approach was originally introduced by Alansary et al. (2016) for placenta segmentation, but was also applied in Cai et al. (2017) to the task of pancreas segmentation. Kamnitsas et al. (2017) proposes another related approach where two parallel networks, a FCN that extracts a rough mask and a HED that outputs a contour, are fused inside a CRF. In the approach by Shen et al. (2018) that deals with brain tumor segmentation, a third path is added where in total three concurrent FCNs are trained based on different filtered (gaussian, mean, median) input images. After each network an individual CRF is applied and their results are fused in a linear regression model.

Figure 2: Overview of relevant papers per year for each category

Training CNN and CRF models end-to-end

The idea of integrating CRF models directly into neural networks origins from the task of semantic image segmentation and was introduced by Zheng et al. (2015). They combine the strengths of both models into a unified framework that allows end-to-end training. Broken down, the basic task of CRFs is to minimize an energy term with an iterative mean field approximation. Since CRFs are graphical models, each iteration step can be formulated as a stack of CNN layers. Multiple iterations can then be implemented by repeatedly executing this stack or alternatively as an equivalent Recurrent Neural Network (RNN). The resulting network is denoted as a CRF-RNN and can be applied on top of any CNN architecture. Fu et al. (2016a) are the first to transfer this method to medical image segmentations with a model called DeepVessel for the task of retinal vessel segmentation. For the same task Luo et al. (2017) achieve similar results by using a slightly deeper base CNN network with more convolution layers. Besides retinal vessel, CRF-RNN approaches are applied to a variety of other anatomical structures. Zhao et al. (2016) applies them to brain tumor segmentation and extend it with some additional pre- and post-processing steps later on Zhao et al. (2018b). Xu et al. (2018) uses a V-Net combined with CRF-RNN for bladder segmentation and in Monteiro et al. (2018) they are also applied on brain tumor as well as prostate segmentation with 3D multi-modal images. Analogues Chen and de Bruijne (2018) utilizes a U-Net as their base-network to deal with white matter lesion segmentation. On the same idea as CRF-RNN Deng et al. (2020) uses a CRF-Recurrent Regression based Neural Network (CRF-RRNN) integrated with a heterogeneous CNN for brain tumor segmentation where the combined network can also be trained end-to-end. Instead of using a full RNN, Zhang et al. (2020d) propose a method where MRF is integrated into the segmentation network as a block of local and global convolution layers that take the CNN output as unary potentials to calculate the corresponding pairwise potentials.

CNNs combined with CRF / MRF models
Authors Anatomy Title Method
\endhead CRF / MRF used for post-processing
Li et al. (2017) Brain Tumor Low-Grade Glioma Segmentation Based on CNN with Fully Connected CRF CRF refines CNN segmentation
Wachinger et al. (2018) Brain Region DeepNAT: Deep convolutional neural network for segmenting neuroanatomy CRF refines hierarchical CNN segmentations
Hu et al. (2019) Brain Tumor Brain Tumor Segmentation Using Multi-Cascaded Convolutional Neural Networks and Conditional Random Field FC-CRF refines segmentations of three CNNs
Shen and Zhang (2017) Brain Tumor Fully connected CRF with data-driven prior for multi-class brain tumor segmentation Multiple FC-CRFs
Kamnitsas et al. (2017) Brain Lesion Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation FC-CRF refines two-pathway CNN
Alansary et al. (2016) Placenta Fast Fully Automatic Segmentation of the Human Placenta from Motion Corrupted MRI FC-CRF refines two-pathway CNN
Shakeri et al. (2016) Sub-cortical regions Sub-cortical brain structure segmentation using F-CNN’s MRF refines FCNN segmentation
Zhai and Li (2019) Brain region An Improved Full Convolutional Network Combined with Conditional Random Fields for Brain MR Image Segmentation Algorithm and its 3D Visualization Analysis FC-CRF refines CNN with attention
Dou et al. (2016) Liver 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes FC-CRF refines 3D FCNN with 3D supervision mechanism
Dou et al. (2017) Heart 3D Deeply Supervised Network for Automated Segmentation of Volumetric Medical Images FC-CRF refines cascading U-Nets
Christ et al. (2016) Liver Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields FC-CRF refines cascaded FCNs
Fu et al. (2016b) Retinal Vessel Retinal vessel segmentation via deep learning network and fully-connected conditional random fields FC-CRF refines CNN with side-outputs
Jin et al. (2018) Left atrial appendage Left Atrial Appendage Segmentation Using Fully Convolutional Neural Networks and Modified Three-Dimensional Conditional Random Fields FC-CRF combines slices of FCN
Cai et al. (2017) Pancreas Pancreas Segmentation in MRI using Graph-Based Decision Fusion on Convolutional Neural Networks CRF refines results from FCN and HED network
Xia et al. (2019) Kidney Deep Semantic Segmentation of Kidney and Space-Occupying Lesion Area Based on SCNN and ResNet Models Combined with SIFT-Flow Algorithm MRF refines combined ResNet and SCNN
Rajchl et al. (2017) Fetal Brain / Lung DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks Iterative CRF and CNN
Nogues et al. (2016) Lymph Node Automatic Lymph Node Cluster Segmentation Using Holistically-Nested Neural Networks and Structured Optimization in CT Images CRF refines HNN (FCN + DSN) segmentations
Yaguchi et al. (2019) Lung Nodules 3D fully convolutional network-based segmentation of lung nodules in CT images with a clinically inspired data synthesis method CRF refines 3D FCN segmentations
Gao et al. (2016) Lung Segmentation label propagation using deep convolutional neural networks and dense conditional random field CRF refines CNN segmentations
Feng et al. (2020) Brain Tumor Study on MRI Medical Image Segmentation Technology Based on CNN-CRF Model CRF refines DCNN segmentations
Liu et al. (2018) Cervical Nuclei Automatic segmentation of cervical nuclei based on deep learning and a conditional random field Locally FC-CRF refines Mask-RCNN segmentation
Shen et al. (2018) Brain Tumor Brain tumor segmentation using concurrent fully convolutional networks and conditional random fields Concurrent FCN refined by FC-CRF
Mesbah et al. (2017) Eye Sclera Conditional random fields incorporate convolutional neural networks for human eye sclera semantic segmentation Initial CNN boundaries refined by CRF
Luo and Yang (2018) Melanoma Fast skin lesion segmentation via fully convolutional network with residual architecture and CRF CRF refines FCN segmentations
Bhatkalkar et al. (2020) Fundus Optic Disk Improving the Performance of Convolutional Neural Network for the Segmentation of Optic Disc in Fundus Images Using Attention Gates and Conditional Random Fields FC-CRF refines CNN segmentations
Qiu et al. (2020) Skin Lesion Inferring Skin Lesion Segmentation With Fully Connected CRFs Based on Multiple Deep Convolutional Neural Networks CRF refines segmentations of DCNN ensemble
Nguyen et al. (2018) Ocular structures Ocular structures segmentation from multi-sequences mri using 3d unet with fully connected crfs FC-CRF refines CNN segmentations
Cao et al. (2019) Prostate cancer lesions Prostate Cancer Detection and Segmentation in Multi-parametric MRI via CNN and Conditional Random Field Selective Dense CRF refines CNN segmentations
CNN and CRF trained end-to-end
Zhao et al. (2018b) Brain Tumor A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Combination of FCNN and CRF-RNN
Monteiro et al. (2018) Prostate / Brain Tumor Conditional Random Fields as Recurrent Neural Networks for 3D Medical Imaging Segmentation Combination of FCNN and CRF-RNN
Fu et al. (2016a) Retinal Vessel DeepVessel: Retinal Vessel Segmentation via Deep Learning and Conditional Random Field Combination of CNN and CRF-RNN layers
Chen and de Bruijne (2018) White matter hyperintensities An End-to-end Approach to Semantic Segmentation with 3D CNN and Posterior-CRF in Medical Images Combination of U-Net and FC-CRF
Xu et al. (2018) Bladder Automatic bladder segmentation from CT images using deep CNN and 3D fully connected CRF-RNN Combination of CNN and CRF-RNN
Deng et al. (2020) Brain Tumor Deep Learning-Based HCNN and CRF-RRNN Model for Brain Tumor Segmentation Combination of HCNN and CRF-RRNN
Zhang et al. (2020d) Prostate ARPM-net: A novel CNN-based adversarial method with Markov Random Field enhancement for prostate and organs at risk segmentation in pelvic CT images CNN combined with MRF block
Zhao et al. (2016) Brain tumor Brain tumor segmentation using a fully convolutional neural network with conditional random fields CRF integrated into FCNN
Luo et al. (2017) Retinal Vessel Efficient CNN-CRF network for retinal image segmentation Combination of CNN and CRF

3 Shape model based approaches

The second category of model assumptions often combined with CNNs are active shape models (ASM) Cootes et al. (1995) or probabilistic active shape models (PASM). ASMs require a training set with a fixed number of manually annotated landmark points of the segmented object. Each point represents a particular part of the object and has to be in the same position over all images. These annotated shapes are then iteratively matched and a mean shape is derived. The landmark points show different variabilities that are modeled by a Point Distribution Model (PDM). Performing a principal component analysis (PCA) and weighting the eigenvectors allows creating new shapes in the allowed variability range. For detecting an object in an unknown image an algorithm is used that updates pose and shape parameters iteratively to improve the match until convergence. An extension to this approach are probabilistic ASMs (PASM) Wimmer et al. (2009). They impose a weaker constraint on shapes which allows more flexible contours with more variations from the mean shape. This is achieved by introducing a probabilistic energy function which is minimized in order to fit a shape to a given image. The model’s ability to generalize is thereby improved and the segmentation results outperform standard ASMs.

Shape Models for post-processing

Though CNN based segmentation models yield good segmentation results, they tend to produce anatomically implausible segmentation maps that can contain detached islands or holes at parts where they do not occur in reality. Since shape models represent valid and anatomically plausible shapes, it makes sense to apply them in post-processing steps to regularize initial CNN segmentations and transform them into a valid shape domain. Xing et al. (2016) take up this idea and apply it to nucleus segmentation. The initial segmentations are generated by a CNN and the post-processing step includes a sparse selection-based shape model for top-down shape inference, which is more insensitive to object occlusions compared to PCA-based shape models, and an additional deformable model for bottom-up shape deformation. Also Hsu (2019) follows this strategy for segmentation and tracking of the left ventricle. They swap out the CNN for a Faster-RCNN and use an improved ASM that allows to obtain matching points in greater ranges. Fauser et al. (2019) continue on improving the ASM by using a probabilistic ASM that is more flexible and allows leaving the shape space. The segmentation of the left ventricle is performed by combining the results of three CNN-PASM models for each dimension. Another modified ASM is proposed by Medley et al. (2020). The authors use Expectation-Maximization to deal with outliers during optimizing the ASM. They also evaluate different ASM features and conclude that a CNN that learns the input feature maps for the EM-ASM performs best. Besides improving on the ASM a different approach by Karimi et al. (2019) aims for generating better predictions with an ensemble of U-Net like CNN models with different filters and parameters. In their approach a SSM model, based on the thresholded segmentations from all individual models, is only applied if the disagreement between the ensemble models becomes to high. Instead of using the CNN for generating segmentation maps, it is also sufficient to only predict bounding boxes as initializations for ASMs. Such an approach is applied by Tabrizi et al. (2018) on kidney segmentation where a fuzzy-ASM produces the final segmentations. Li et al. (2018) also uses a CNN for bounding box prediction, but adds an intermediate step before utilizing a statistical shape model for myocardial segmentation, in which a random forest classifier builds probability maps from the given bounding boxes. Another tree model, more specific an adaptive feature learning probability boosting tree (AFL-PBT) is also utilized by He et al. (2018) as an initial step to classify voxels for prostate segmentation. A subsequent CNN then extracts boundary probability maps and a three-level ASM is employed to generate final segmentations.

Shape Models for prior knowledge

In this second paragraph we present some papers where the shape models are applied pre-hoc before any deep learning network. Two straight forward models for this category are proposed by Cheng et al. (2016) and Fan et al. (2020). In Fan et al. (2020) a 3D U-Net-like CNN segments Itra-Cholear anatomy based on initial segmentations from an ASM and the original CT images. Cheng et al. (2016) on the other hand use a CNN for refining initial segmentations from an Active Appearance Model (AAM) that produces only coarse prostate segmentations. The AAM is basically an extended shape model that adds an additional texture model for better fitting capabilities. The other two models already introduce some pipeline-like approaches, but use both a shape model as prior knowledge. The pipeline for subcortical region segmentation in Duy et al. (2018) starts with a pre-processing SVM that classifies sagittal slices into groups of similar shape. The prior ASM then creates rough segmentations for each group which are finalized by a CNN. Further the authors propose an optional CRF model for post-processing. Nguyen et al. (2019) introduce the ASM as a more traditional prior for uveal melanoma segmentation where it is used as a constraining term for a CRF model that is based on Grad-CAM (class activation maps) heatmaps. The final segmentations are again generated with a U-Net that combines the CRF with original input CTs.

Figure 3: Overview of anatomical structures examined in the relevant papers

Pipeline approaches with multiple CNN and ASM models

The last category for combining shape models and neural networks contains all approaches that consist of different models arranged along pipelines. The motivation is to process input images stage-wise or in a coarse-to-fine way that allows to capture more information and hence result in more accurate segmentation maps. In the models by Tack et al. (2018) for knee menisci, Ambellan et al. (2019) for knee bone & cartilage, and Brusini et al. (2020) for hippocampus segmentation, the pipelines combine multiple CNNs and SSMs. All three start with initial 2D U-Nets regularized by SSMs which are used to extract smaller 3D subvolumes. Tack et al. (2018) and Ambellan et al. (2019) apply an additional 3D U-Net afterwards, whereas Brusini et al. (2020) uses three U-Nets and averages their predictions to obtain final segmentations. Ambellan et al. (2019) further continues after this step and utilizes a second 3D SSM model to obtain the knee bone segmentations and even applies a third U-Net to segment the cartilage afterwards. Besides these typical pipelines, there are also some hybrid approaches we count to this category that integrate shape models and neural networks. They use special CNNs that directly predicts the parameters of an SSM, which are the shape coefficients (weights for the modes of variations), the pose parameters. Qin et al. (2020) use such a SSM-Net inside a small pipeline for prostate segmentation. They propose an inception-based network that directly predicts parameters of the SSM which can be back-translated into a prostate contour prediction. Parallel to this, a residual U-Net generates probability maps from the inputs. The final segmentations are generated by averaging the outputs of both models. The method of Tilborghs et al. (2020) for left ventricle segmentation is based on the same idea, but removes the small pipeline. Instead they modify the CNN and add a third output which is an actual distance map. A special loss function is used to train the network toward optimizing the segmentation map alongside the SSM parameters. A nearly identical approach by Karimi et al. (2018) is applied to prostate segmentation. Their CNN predicts center position of the prostate, the shape model parameters, and a rotation vector which are passed to a final layer that outputs the coordinates of the landmark points which resemble the a final segmentation map. Schock et al. (2020) relies on the same method for knee bone & cartilage segmentation, but extend it with additional pre- and post-processing steps. They add a preprocessing 2D U-Net that detects initial bone positions and crop the volume into subvolumes which only contain the femur or tibia bone. Afterwards their SSM-Net comes into place that predicts the SSM parameters and the actual landmarks in a subsequent PCA layer. An additional fine-tuning step then generates the cartilage segmentations with a 3D U-Net based on subvolumes centered at the bones’ landmark points. Rather than integrating the SSM and CNN, Ma et al. (2018) introduces a Bayesian model that integrates both, the CNN and a robust kernel SSM (RKSSM) for the task of pancreas segmentation. At first the RKSSM is initialized to fit the detected ROI of a Dense U-Net. A Gaussian Mixture Model afterwards guides the shape adaption and iteratively projects the adapted shape onto the RKSSM until convergence which results in the final segmentation map.

CNNs combined with Active Shape Models
Authors Anatomy Title Method
\endhead ASM for post-processing
Xing et al. (2016) Nucleus An Automatic Learning-Based Framework for Robust Nucleus Segmentation Shape Model refines CNN segmentation
He et al. (2018) Prostate Automatic Magnetic Resonance Image Prostate Segmentation Based on Adaptive Feature Learning Probability Boosting Tree Initialization and CNN-ASM Refinement Three-level-ASM refines segmentations of CNN
Fauser et al. (2019) Temporal Bone Toward an automatic preoperative pipeline for image-guided temporal bone surgery Probabilistic ASM refines 2D U-Net segmentation
Li et al. (2018) Myocardial Fully Automatic Myocardial Segmentation of Contrast Echocardiography Sequence Using Random Forests Guided by Shape Model ASM refines random-forest segmentations initialized by a CNN
Medley et al. (2020) Left Ventricle Deep Active Shape Model for Robust Object Fitting ASM initialized with CNN generated features maps
Karimi et al. (2019) Prostate Accurate and robust deep learning-based segmentation of the prostate clinical target volume in ultrasound images SSM refines segmentations from ensemble of CNNs
Tabrizi et al. (2018) Kidney Automatic kidney segmentation in 3D pediatric ultrasound images using deep neural networks and weighted fuzzy active shape model Fuzzy ASM segmentations based on DNN generated bounding boxes
Hsu (2019) Left Ventricle Automatic Left Ventricle Recognition, Segmentation and Tracking in Cardiac Ultrasound Image Sequences ASM improves R-CNN segmentations for detection and tracking
ASM as prior-knowledge
Duy et al. (2018) Brain Region Accurate brain extraction using Active Shape Model and Convolutional Neural Networks CNN refines ASM segmentations
Cheng et al. (2016) Prostate Active appearance model and deep learning for more accurate prostate segmentation on MRI 2D-CNN refines segmentations from an Active Appearance Model
Fan et al. (2020) Intra-Cholear Anatomy Combining model- and deep-learning-based methods for the accurate and robust segmentation of the intra-cochlear anatomy in clinical head CT images U-Net refines ASM segmentations
Nguyen et al. (2019) Uveal Melanoma A novel segmentation framework for uveal melanoma based on magnetic resonance imaging and class activation maps U-Net segmentations based on a CRF that uses ASM as prior knowledge
Pipelines with multiple ASM and CNN models & Hybrid approaches
Ambellan et al. (2019) Knee Bone / Cartilage Automated Segmentation of Knee Bone and Cartilage combining Statistical Shape Knowledge and Convolutional Neural Networks: Data from the Osteoarthritis Initiative Three CNN and two SSM models
Tack et al. (2018) Knee Menisci Knee Menisci Segmentation using Convolutional Neural Networks: Data from the Osteoarthritis Initiative 3D CNN and SSM initialized by 2D models
Brusini et al. (2020) Hippocampus Shape Information Improves the Cross-Cohort Performance of Deep Learning-Based Segmentation of the Hippocampus ASM as input for CNN
Ma et al. (2018) Pancreas A novel bayesian model incorporating deep neural network and statistical shape model for pancreas segmentation U-Net and SSM segmentations combined within Bayesian model
Qin et al. (2020) Prostate A weakly supervised registration-based framework for prostate segmentation via the combination of statistical shape model and CNN Segmentations combined of U-Net and SSM-Net predictions
Tilborghs et al. (2020) Left Ventricle Shape Constrained CNN for Cardiac MR Segmentation with Simultaneous Prediction of Shape and Pose Parameters Hybrid approach where CNN generates segmentations and ASM parameters
Karimi et al. (2018) Prostate Prostate segmentation in MRI using a convolutional neural network architecture and training strategy based on statistical shape models CNN predicts segmentations and 3D-ASM parameters
Schock et al. (2020) Knee Bone & Cartilage A Method for Semantic Knee Bone and Cartilage Segmentation with Deep 3D Shape Fitting Using Data from the Osteoarthritis Initiative CNN that predicts segmentations and 3D-ASM parameters is refined by U-Net

4 Active contour approaches

A last type of models that often combined with deep learning models to incorporate shape knowledge are Active Contour Models (ACM) Kass et al. (1988) , also known as snakes. A snake is a deformable controlled continuity spline that is pushed towards edges or contours by minimizing an energy function under the influence of different forces and constraints. It consists of an internal energy that keeps the contour continuous and smooth, an image energy that attracts it to contours, and an external constraint force that adds user-imposed guidance. A similar approach are level set functions (LSF) introduced by Andrew (2000) and firstly applied to image segmentation by Malladi et al. (1995). An LSF is a higher dimensional function where a contour is defined as its zero level set. With a speed function, derived from the image, that controls the evolution of the surface over time, a Hamilton-Jacobi partial differential equation can be obtained.

ACM models for post-processing

Since ACM models are based on the idea of evolving a contour, it makes sense to apply them as a post-processing step to improve an initial segmentation map. An early model by Middleton and Damper (2004) uses only a simple multilayer perceptron (MLP) that creates binary pixel-wise boundary predictions for lung segmentation. Since these are very rough and contain misclassifications the ASM is used to improve and close the contour. Salimi et al. (2018) is also based on an MLP, but adds an vector field convolution to the ACM to make it more robust for prostate segmentation.
However, the more recent ACM post-processing models are exclusively based on different CNN architectures and are applied to a variety of anatomies. Li et al. (2017) use a FCN that is refined by a classic ACM for left ventricle segmentation. The same approach is taken by Guo et al. (2019) for liver segmentation and Zhao et al. (2018a) utilize it for nucleus segmentation. In the approaches by Xu et al. (2019) the ACM refinements are not yet the final steps and additional adaptive ellipse fitting is used to segment breast nuclei. Hu et al. (2018) and Fang et al. (2019) transfer the basic refinement method to breast tumor detection with a phase-based ACM that improves over multiple iterations. A different slightly modified ACM post-processing method is based on geodesic computations and is further used by Ma and Yang (2019) for dental root segmentation and Nunes et al. (2020) for lung segmentation. Zhang et al. (2020b) also introduces a special ACM that integrates a fourth-order partial differential equation and segments plaque based on an initial R-CNN segmentation. Instead of just refining an initial CNN predicted per-pixel segmentation map, da Silva et al. (2020) use a Chan-Vese ACM to generate prostate segmentation on DCNN coarsely classified superpixels which only represent rough initialization for the contour model. The authors of Kot et al. (2020) further separate the two models where the CNN masks bone tissue which is removed for the ACM to segment brain tumors. The last special approach in the ACM category by Zhang et al. (2020c) is a hybrid model that integrates an ACM into a U-Net. The resulting deep active contour network (DACN) is end-to-end trainable with a special ACM based loss function and automatically segments cervical cells and skin lesions. Besides ACM, another large number of approaches rely on level set functions (LSF). Same as before a CNN is used for generating initial segmentation maps which are then refined by the LSF. Hatamizadeh et al. (2019) uses this for brain, liver, lung segmentation, Gong et al. (2019) for pancreas segmentation, Carbajal-Degante et al. (2020) for ventricle and liver segmentation, and Xie et al. (2020) for left ventricle segmentation. Some extra processing is made in Yang et al. (2021) for dental pulp segmentation where the initial CNN segmentations are used to calculate elliptic curves which are used to guide the LSF. In general, for the LSF it is often sufficient to initialize them only with a rough bounding boxes or region of interest annotations. So, Liu et al. (2019) use a Faster RCNN to generate location boxes of left atriums which serve as input for the LSF after Otsu thresholding. Avendi et al. (2016) inserts an additional step between CNN ROI detection and LSF segmentation where the initial left-ventricle shape is inferred with an stacked auto-encoder. In comparison to these two approaches, in Cha et al. (2016) the CNN is not used to predict ROI, but to classify if an ROI is part of the bladder. The outputs are then refined by three different 3D LSF and a final 2D LSF afterwards. Another idea is to use recurrent pipelines where the segmentations are refined iteratively. Such an approach is introduced by Tang et al. (2017) where both models are integrated into a FCN-LSF. The method is used for left ventricle and liver segmentation with semi-supervised training where the LSF gradually refines the segmentation and backpropagates a loss to improve the FCN. Hoogi et al. (2017) proposed a different iterative process. Hereby the CNN estimates if the zero level set is inside, outside or near the lesion boundary. Based on these the LSF parameters are calculated and the contour is evolved. The process then repeats until convergence.

Using a CNN to refine ACM segmentations

Besides the majority of approaches that use ACMs for post-processing, there are also methods where ACMs are used to obtain the initial segmentations or are guided by CNNs. The earliest of these approaches by Ahmed et al. (2009) uses an ACM to remove skull tissue from images and applies a simple artificial neural network to classify the remaining brain regions. Rupprecht et al. (2016) introduce an approach where the ACM is guided by the CNN. The ACM generated rough segmentations of the left ventricle. A CNN then predicts vectors on patches around each pixel of this initial contour that point towards closes object boundary points and are used to further evolve the contour. The latest method for this category by Kasinathan et al. (2019) also uses the ACM to generate initial segmentations, more specific it segments all lung nodules. A post-processing CNN afterwards classifies them or removes false positives.

CNNs combined with Active Contour Models
Authors Anatomy Title Method
\endhead ACM for post-processing
Middleton and Damper (2004) Lung Segmentation of magnetic resonance images using a combination of neural networks and active contour models ACM refines MLP segmentation
Salimi et al. (2018) Prostate Fully automatic prostate segmentation in MR images using a new hybrid active contour-based approach ACM refines MLP segmentation
Li et al. (2017) Left Ventricle Left ventricle segmentation by combining convolution neural network with active contour model and tensor voting in short-axis MRI ACM refines FCN segmentation
Hu et al. (2018) Breast Tumor Automatic tumor segmentation in breast ultrasound images using a dilated fully convolutional network combined with an active contour model Phase-based ACM refines dilated FCN segmentation
Guo et al. (2019) Liver Automatic liver segmentation by integrating fully convolutional networks into active contour models ACM refines multi-branch FCN segmentation
Zhao et al. (2018a) Nucleus Improved Nuclear Segmentation on Histopathology Images Using a Combination of Deep Learning and Active Contour Model Hybrid ACM refines multi-branch FCN segmentation
Hatamizadeh et al. (2019) Liver / Brain Lesion / Lung Deep Active Lesion Segmentation ACM refines signed distance maps from FC-CNN
Tang et al. (2017) Liver / Left Ventricle A Deep Level Set Method for Image Segmentation Level-set ACM refines FCN segmentations iteratively
Cha et al. (2016) Bladder Urinary bladder segmentation in CT urography using deep-learning convolutional neural network and level sets Multiple level-set functions segment CNN output ROIs
Hoogi et al. (2017) Liver Lesion Adaptive Estimation of Active Contour Parameters Using Convolutional Neural Networks and Texture Analysis Level-set function iteratively improves CNN segmentation
Fang et al. (2019) Breast Tumor Combining a Fully Convolutional Network and an Active Contour Model for Automatic 2D Breast Tumor Segmentation from Ultrasound Images Phase-based ACM refines initial contours from dilated FCNN
Xu et al. (2019) Breast Cancer Nuclei Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images ACM refines CNN segmentations
Ma and Yang (2019) Teeth Automatic dental root CBCT image segmentation based on CNN and level set method ACM refines CNN segmentations
Carbajal-Degante et al. (2020) Ventricles Active contours for multi-region segmentation with a convolutional neural network initialization Phase level-set function refines CNN segmentations
Liu et al. (2019) Left Atrium A Framework for Left Atrium Segmentation on CT Images with Combined Detection Network and Level Set Model 3D level-set model initialized by Faster RCNN
Yang et al. (2021) Teeth Accurate and automatic tooth image segmentation model with deep convolutional neural networks and level set method Level-set based on contours derived from U-Net predictions
Nunes et al. (2020) Lung Adaptive Level Set with region analysis via Mask R-CNN: A comparison against classical methods ACM improves Mask R-CNN segmentations
Xie et al. (2020) Left Ventricle Automatic left ventricle segmentation in short-axis MRI using deep convolutional neural networks and central-line guided level set approach Level-set model improves CNN initialization
Gong et al. (2019) Pancreas Convolutional Neural Networks Based Level Set Framework for Pancreas Segmentation from CT Images Level-set model based on initial contour from CNN
Zhang et al. (2020c) Cervical Cell / Skin Lesion Deep Active Contour Network for Medical Image Segmentation ACM integrated into CNN that learns initial parameters (end-to-end)
Zhang et al. (2020b) Plaque Faster R-CNN, fourth-order partial differential equation and global-local active contour model (FPDE-GLACM) for plaque segmentation in IV-OCT image ACM initialized with bounding box from R-CNN
da Silva et al. (2020) Prostate Superpixel-based deep convolutional neural networks and active contour model for automatic prostate segmentation on 3D MRI scans ACM refines DCNN segmentations
Kot et al. (2020) Brain Tumor U-Net and Active Contour Methods for Brain Tumour Segmentation and Visualization ACM refines U-Net segmentations
Avendi et al. (2016) Left Ventricle A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI CNN and AE initialize level set function
CNN refines ACM
Kasinathan et al. (2019) Lung Tumor / Nodule Automated 3-D Lung Tumor Detection and Classification by an Active Contour Model and CNN Classifier CNN refines multiple ACM segmentations
Rupprecht et al. (2016) Left ventricular cavity Deep Active Contour CNN refines ACM
Ahmed et al. (2009) Brain A Hybrid Approach for Segmenting and Validating T1-Weighted Normal Brain MR Images by Employing ACM and ANN ANN based on ACM preprocessed images

5 Topology based Approaches

An alternative approach to integrating shape priors into network-based segmentation was presented in Lee et al. (2019). Here, the segmentation started with a candidate shape which was topologically correct (and approximately correct in terms of its shape), and the network was trained to provide the appropriate deformation to this shape such that it maximally overlapped with the ground truth segmentation.

Such methods can be considered to have a ‘hard prior’ rather than the ‘soft-prior’ of the methods presented above in the sense that the end result can be guaranteed to have the correct shape. However, this approach may be limited by a requirement that the initial candidate shape be very close to an acceptable answer such that only small shape deformations are needed. A further potential issue is that the deformation field provided by the network may need to be restricted to prevent the shape from overlapping itself and consequently changing its topology.

The differentiable properties of persistent homology Edelsbrunner et al. (2000) make it a promising candidate for the integration of topological information into the training of neural networks. The key idea is that it measures the presence of topological features as some threshold or length scale changes. Persistent features are those which exist for a wide range of filtration values, and this persistence is differentiable with respect to the original data. There have recently been a number of approaches suggested for the integration of PH and deep learning, which we briefly review here.

In Chen et al. (2018) a classification task was considered, and PH was used to regularise the decision boundary. Typical regularisation of a decision boundary might encourage it to be smooth or to be far from the data. Here, the boundary was encouraged to be simple from a topological point of view, meaning that topological complexities such as loops and handles in the decision boundary were discouraged. Rieck et al. (2018) proposed a measure of the complexity of a neural network using PH. This measure of ‘neural persistence’ was evaluated as a measure of structural complexity at each layer of the network, and was shown to increase during network training as well as being useful as a stopping criterion.

PH is applied to image segmentation, but the PH calculation has typically been applied to the input image and used as a way to generate features which can then be used by another algorithm. Applications have included tumour segmentation Qaiser et al. (2016), cell segmentation Assaf et al. (2017) and cardiac segmentation from computed tomography (CT) imaging Gao et al. (2013). Recently Clough et al. (2019a) proposed to use PH not to the input image being segmented, but rather to the candidate segmentation provided by the network. In an extended work Clough et al. Clough et al. (2020) the topological information found by the PH calculation can be used to provide a training signal to the network, allowing an differentiable loss function to compare the topological features present in a proposed segmentation, with those specified to exist by some prior knowledge.

6 Discussion

As the deep learning research effort for medical image segmentation is consolidating towards incorporating shape constraints to ensure downstream analysis, certain patterns are emerging as well. In the next few subsections, we discuss such clear patterns and emerging questions relevant for the progress of research in this direction.

6.1 End-to-End vs post/pre-hoc

With the maturity of research, this field is clearly moving beyond post-/pre-hoc setting towards more systematic end-to-end training approaches. This effect is depicted in Figure 4 where the paper counts are aggregated from this work and Jurdi et al. (2020). The maturity of deep learning frameworks (especially PyTorch), novel architectures (especially generative modeling) and automatic differentiation make it possible to incorporate complex shape-based loss functions during training. With the availability of these tools, large models can be trained with tailored shape streams in the model architecture to incorporate shape information.

Figure 4: Temporal trend towards end-to-end approaches

6.2 Semi-supervised segmentation

The ability to incorporate additional information using shape as a prior can aid in reducing the total number of necessary annotations in achieving a good segmentation. The shape priors can useful in generating controlled data augmentations for the medical image analysis task in hand and reduce the number of unrealistic augmentations. This would be instrumental in particular in the case of rare diseases, where there is not enough of data and manual annotations to train a neural network. The shape priors that are giving clues about the expected pathology in such cases can lead to better segmentation accuracy in the final output.

6.3 Effectiveness in pathological cases

One common theme identified by last few decades worth research on shape modeling is the difficulty in representing the pathological shapes. While the “typical shapes” i.e. normal shapes lie in a low-dimensional sub-manifold, the pathological cases have a long tail in the distribution (e.g. congenital heart diseases). That is normal shapes are self-similar but pathological cases contain atypical shapes along with typical pathologies. Traditional linearized shape modeling had trouble addressing this issue whereas the non-linear modeling of shape statistics had its issue in terms of intractable numerics. Whether a neural approach can address this overarching problem of encoding pathological shapes is an open problem. Unfortunately, from our literature search, we have not found any clear direction to address this perennial issue of shape modeling.

6.4 Evaluation

While the shape constraints are becoming increasingly commonplace for medical image segmentation, we believe the visual perception and human comprehension plays a significant role behind the interest of the community. The more general question of real world effectiveness of these methods are not often studied. For example, how effective these shape constraints are under noisy annotation is an open question? While the segmentation quality is most often measured by the Dice metric, Maier-Hein et al. (2018) has already prescribed to move beyond Dice to evaluate the segmentation quality. Topological accuracy of anatomical structures is increasingly used as an evaluation metric to address the shortcomings of classical image segmentation evaluation metric in medical image analysis Byrne et al. (2020). Finally, segmentation is typically a mean to an end. As such, the effectiveness of these segmentation techniques should be measured quantitatively for downstream evaluation tasks such as visualization, planning Fauser et al. (2019) etc.

7 Conclusion

Bringing prior knowledge about the shape of the anatomy for semantic segmentation is a rather well-trodden idea. The community is devising new ways to incorporate such prior knowledge in deep learning models trained with frequentist approach. While the Bayesian interpretation of deep learning segmentation networks is an upcoming trend, it is already shown that under careful considerations, prior knowledge about the shape can be incorporated even in frequentist approaches with significant success.

We see the future research concentrating more on end-to-end networks with the overarching theme of learning using Analysis-by-synthesis. Early work has demonstrated the effectiveness of shape constraints in federated learning and this will be a major direction in the coming years.

We believe the community needs to address the issues discussed in Section 6 before shape constrained segmentation can be considered as a trustworthy technology in practical medical image analysis. To this end, we can think of shape constrained segmentation as a technical building block within a bigger image analysis pipeline rather than a stand-alone piece of technology. For example, in the case of surgical planning and navigation pipeline, such shape constraints can be meaningful provided the performance is thoroughly validated under pathological cases with multiple quality metrics. Important steps have already been taken in this direction. In short, along with exciting results, shape constrained deep learning for segmentation opens up many possible research questions for the next few years. Proper understanding and answering those hold the key to their successful deployment in the real clinical scenario.

References

  1. A hybrid approach for segmenting and validating t1-weighted normal brain MR images by employing ACM and ANN. In First International Conference of Soft Computing and Pattern Recognition, SoCPaR 2009, Malacca, Malaysia, December 4-7, 2009, A. Abraham, A. K. Muda, N. S. Herman, S. M. Shamsuddin and Y. Choo (Eds.), pp. 239–244. External Links: Link, Document Cited by: §4, §4.
  2. Fast fully automatic segmentation of the human placenta from motion corrupted MRI. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II, pp. 589–597. External Links: Link, Document Cited by: §2, §2.
  3. Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: data from the osteoarthritis initiative. Medical Image Analysis 52, pp. 109–118. External Links: Link, Document Cited by: §3, §3.
  4. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science, by J.A. sethian, cambridge university press, cambridge, uk, 2nd edn 1999 (first published 1996 as Level Set Methods) xviii + 420 pp., ISBN (paperback) 0-521-64557-3, (hardback) 0-521-64204-3 (pbk, £18.95). Robotica 18 (1), pp. 89–92. External Links: Link Cited by: §4.
  5. Topological persistence based on pixels for object segmentation in biomedical images. In 2017 Fourth International Conference on Advances in Biomedical Engineering (ICABME), pp. 1–4. Cited by: §5.
  6. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Medical Image Anal. 30, pp. 108–119. External Links: Link, Document Cited by: §4, §4.
  7. Improving the performance of convolutional neural network for the segmentation of optic disc in fundus images using attention gates and conditional random fields. IEEE Access 8 (), pp. 29299–29310. External Links: Document Cited by: §2, §2.
  8. Shape information improves the cross-cohort performance of deep learning-based segmentation of the hippocampus. Frontiers in Neuroscience 14, pp. 15. External Links: Link, Document, ISSN 1662-453X Cited by: §3, §3.
  9. A persistent homology-based topological loss function for multi-class cnn segmentation of cardiac mri. arXiv preprint arXiv:2008.09585. Cited by: §6.4.
  10. Pancreas segmentation in MRI using graph-based decision fusion on convolutional neural networks. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2017 - 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III, pp. 674–682. External Links: Link, Document Cited by: §2, §2.
  11. Prostate cancer detection and segmentation in multi-parametric MRI via CNN and conditional random field. In 16th IEEE International Symposium on Biomedical Imaging, ISBI 2019, Venice, Italy, April 8-11, 2019, pp. 1900–1904. External Links: Link, Document Cited by: §2, §2.
  12. Active contours for multi-region segmentation with a convolutional neural network initialization. In Optics, Photonics and Digital Technologies for Imaging Applications VI, P. Schelkens and T. Kozacki (Eds.), Vol. 11353, pp. 36 – 44. External Links: Document, Link Cited by: §4, §4.
  13. Urinary bladder segmentation in ct urography using deep-learning convolutional neural network and level sets. Medical Physics 43, pp. 1882–1896. External Links: Document, Link Cited by: §4, §4.
  14. TopoReg: A Topological Regularizer for Classifiers. arXiv 1806.10714. Cited by: §5.
  15. An end-to-end approach to semantic segmentation with 3d CNN and posterior-crf in medical images. CoRR abs/1811.03549. External Links: Link, 1811.03549 Cited by: §2, §2.
  16. Active appearance model and deep learning for more accurate prostate segmentation on MRI. In Medical Imaging 2016: Image Processing, San Diego, California, USA, February 27, 2016, pp. 97842I. External Links: Link, Document Cited by: §3, §3.
  17. Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3d conditional random fields. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II, pp. 415–423. External Links: Link, Document Cited by: §2, §2.
  18. A topological loss function for deep-learning based image segmentation using persistent homology. IEEE Transactions on Pattern Analysis and Machine Intelligence (), pp. 1–1. Cited by: §5.
  19. Explicit topological priors for deep-learning based image segmentation using persistent homology. In International Conference on Information Processing in Medical Imaging, pp. 16–28. Cited by: §5.
  20. A topological loss function for deep-learning based image segmentation using persistent homology. CoRR abs/1910.01877. External Links: Link, 1910.01877 Cited by: §1.1.
  21. Active shape models-their training and application. Computer Vision and Image Understanding 61 (1), pp. 38–59. External Links: Link, Document Cited by: §3.
  22. Superpixel-based deep convolutional neural networks and active contour model for automatic prostate segmentation on 3d MRI scans. Medical Biol. Eng. Comput. 58 (9), pp. 1947–1964. External Links: Link, Document Cited by: §4, §4.
  23. Deep learning-based HCNN and CRF-RRNN model for brain tumor segmentation. IEEE Access 8, pp. 26665–26675. External Links: Link, Document Cited by: §2, §2.
  24. 3D deeply supervised network for automatic liver segmentation from CT volumes. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II, pp. 149–157. External Links: Link, Document Cited by: §2, §2.
  25. 3D deeply supervised network for automated segmentation of volumetric medical images. Medical Image Analysis 41, pp. 40–54. External Links: Link, Document Cited by: §2, §2.
  26. Accurate brain extraction using active shape model and convolutional neural networks. CoRR abs/1802.01268. External Links: Link, 1802.01268 Cited by: §3, §3.
  27. Topological persistence and simplification. In Foundations of Computer Science, pp. 454–463. Cited by: §5.
  28. Medical image segmentation: a brief survey. In Multi Modality State-of-the-Art Medical Image Segmentation and Registration Methodologies: Volume II, A. S. El-Baz, R. Acharya U, A. F. Laine and J. S. Suri (Eds.), pp. 1–39. External Links: ISBN 978-1-4419-8204-9, Document, Link Cited by: §1.1.
  29. Combining model- and deep-learning-based methods for the accurate and robust segmentation of the intra-cochlear anatomy in clinical head CT images. In Medical Imaging 2020: Image Processing, Houston, TX, USA, February 15-20, 2020, I. Isgum and B. A. Landman (Eds.), SPIE Proceedings, Vol. 11313, pp. 113131D. External Links: Link, Document Cited by: §3, §3.
  30. Combining a fully convolutional network and an active contour model for automatic 2d breast tumor segmentation from ultrasound images. Journal of Medical Imaging and Health Informatics 9, pp. 1510–1515. External Links: Document, Link Cited by: §4, §4.
  31. Toward an automatic preoperative pipeline for image-guided temporal bone surgery. Int. J. Comput. Assist. Radiol. Surg. 14 (6), pp. 967–976. External Links: Link, Document Cited by: §3, §3, §6.4.
  32. Study on MRI medical image segmentation technology based on CNN-CRF model. IEEE Access 8, pp. 60505–60514. External Links: Link, Document Cited by: §2, §2.
  33. DeepVessel: retinal vessel segmentation via deep learning and conditional random field. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II, pp. 132–139. External Links: Link, Document Cited by: §2, §2.
  34. Retinal vessel segmentation via deep learning network and fully-connected conditional random fields. In 13th IEEE International Symposium on Biomedical Imaging, ISBI 2016, Prague, Czech Republic, April 13-16, 2016, pp. 698–701. External Links: Link, Document Cited by: §2, §2.
  35. Segmenting the papillary muscles and the trabeculae from high resolution cardiac CT through restoration of topological handles. In IPMI, pp. 184–195. Cited by: §5.
  36. Segmentation label propagation using deep convolutional neural networks and dense conditional random field. In 13th IEEE International Symposium on Biomedical Imaging, ISBI 2016, Prague, Czech Republic, April 13-16, 2016, pp. 1265–1268. External Links: Link, Document Cited by: §2, §2.
  37. Convolutional neural networks based level set framework for pancreas segmentation from CT images. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine, ISICDM 2019, Xi’an, China, August 24-26, 2019, pp. 27–30. External Links: Link, Document Cited by: §4, §4.
  38. Automatic liver segmentation by integrating fully convolutional networks into active contour models. Medical Physics, pp. . External Links: Document, Link Cited by: §4, §4.
  39. Noise-robust pupil center detection through cnn-based segmentation with shape-prior loss. IEEE Access 8, pp. 64739–64749. External Links: Link, Document Cited by: §1.1.
  40. Deep active lesion segmentation. CoRR abs/1908.06933. External Links: Link, 1908.06933 Cited by: §4, §4.
  41. Automatic magnetic resonance image prostate segmentation based on adaptive feature learning probability boosting tree initialization and CNN-ASM refinement. IEEE Access 6, pp. 2005–2015. External Links: Link, Document Cited by: §3, §3.
  42. Statistical shape models for 3d medical image segmentation: A review. Medical Image Analysis 13 (4), pp. 543–563. External Links: Link, Document Cited by: §1.1.
  43. Deep learning techniques for medical image segmentation: achievements and challenges. J. Digit. Imaging 32 (4), pp. 582–596. External Links: Link, Document Cited by: §1.1.
  44. Adaptive estimation of active contour parameters using convolutional neural networks and texture analysis. IEEE Trans. Med. Imaging 36 (3), pp. 781–791. External Links: Link, Document Cited by: §4, §4.
  45. Automatic left ventricle recognition, segmentation and tracking in cardiac ultrasound image sequences. IEEE Access 7, pp. 140524–140533. External Links: Link, Document Cited by: §3, §3.
  46. Brain tumor segmentation using multi-cascaded convolutional neural networks and conditional random field. IEEE Access 7, pp. 92615–92629. External Links: Link, Document Cited by: §2, §2.
  47. Automatic tumor segmentation in breast ultrasound images using a dilated fully convolutional network combined with an active contour model. Medical Physics 46, pp. . External Links: Document, Link Cited by: §4, §4.
  48. Left atrial appendage segmentation using fully convolutional neural networks and modified three-dimensional conditional random fields. IEEE J. Biomedical and Health Informatics 22 (6), pp. 1906–1916. External Links: Link, Document Cited by: §2, §2.
  49. High-level prior-based loss functions for medical image segmentation: A survey. CoRR abs/2011.08018. External Links: Link, 2011.08018 Cited by: §1.1, §6.1.
  50. Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36, pp. 61–78. External Links: Link, Document Cited by: §2, §2.
  51. Prostate segmentation in MRI using a convolutional neural network architecture and training strategy based on statistical shape models. Int. J. Comput. Assist. Radiol. Surg. 13 (8), pp. 1211–1219. External Links: Link, Document Cited by: §3, §3.
  52. Accurate and robust deep learning-based segmentation of the prostate clinical target volume in ultrasound images. Medical Image Anal. 57, pp. 186–196. External Links: Link, Document Cited by: §3, §3.
  53. Automated 3-d lung tumor detection and classification by an active contour model and CNN classifier. Expert Syst. Appl. 134, pp. 112–119. External Links: Link, Document Cited by: §4, §4.
  54. Snakes: active contour models. International Journal of Computer Vision 1 (4), pp. 321–331. External Links: Link, Document Cited by: §4.
  55. U-net and active contour methods for brain tumour segmentation and visualization. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, pp. 1–7. External Links: Link, Document Cited by: §4, §4.
  56. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pp. 282–289. Cited by: §2.
  57. TETRIS: template transformer networks for image segmentation with shape priors. IEEE transactions on medical imaging. Cited by: §5.
  58. Medical image segmentation using deep learning: A survey. CoRR abs/2009.13120. External Links: Link, 2009.13120 Cited by: §1.1.
  59. Joint left atrial segmentation and scar quantification based on a DNN with spatial encoding and shape attention. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 - 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV, A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu and L. Joskowicz (Eds.), Lecture Notes in Computer Science, Vol. 12264, pp. 118–127. External Links: Link, Document Cited by: §1.1.
  60. Markov random field models in computer vision. In Computer Vision - ECCV’94, Third European Conference on Computer Vision, Stockholm, Sweden, May 2-6, 1994, Proceedings, Volume II, pp. 361–370. External Links: Link, Document Cited by: §2.
  61. Fully automatic myocardial segmentation of contrast echocardiography sequence using random forests guided by shape model. IEEE Trans. Med. Imaging 37 (5), pp. 1081–1091. External Links: Link, Document Cited by: §3, §3.
  62. Low-grade glioma segmentation based on cnn with fully connected crf. Journal of Healthcare Engineering 2017, pp. 1–12. External Links: Document, Link Cited by: §2, §2.
  63. Left ventricle segmentation by combining convolution neural network with active contour model and tensor voting in short-axis MRI. In 2017 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2017, Kansas City, MO, USA, November 13-16, 2017, pp. 736–739. External Links: Link, Document Cited by: §4, §4.
  64. A survey on deep learning in medical image analysis. Medical Image Anal. 42, pp. 60–88. External Links: Link, Document Cited by: §1.1.
  65. A framework for left atrium segmentation on CT images with combined detection network and level set model. In 46th Computing in Cardiology, CinC 2019, Singapore, September 8-11, 2019, pp. 1–4. External Links: Link, Document Cited by: §4, §4.
  66. Automatic segmentation of cervical nuclei based on deep learning and a conditional random field. IEEE Access 6, pp. 53709–53721. External Links: Link, Document Cited by: §2, §2.
  67. Fast skin lesion segmentation via fully convolutional network with residual architecture and CRF. In 24th International Conference on Pattern Recognition, ICPR 2018, Beijing, China, August 20-24, 2018, pp. 1438–1443. External Links: Link, Document Cited by: §2, §2.
  68. Efficient cnn-crf network for retinal image segmentation. In Cognitive Systems and Signal Processing, F. Sun, H. Liu and D. Hu (Eds.), Singapore, pp. 157–165. External Links: ISBN 978-981-10-5230-9 Cited by: §2, §2.
  69. A novel bayesian model incorporating deep neural network and statistical shape model for pancreas segmentation. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV, A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López and G. Fichtinger (Eds.), Lecture Notes in Computer Science, Vol. 11073, pp. 480–487. External Links: Link, Document Cited by: §3, §3.
  70. Automatic dental root CBCT image segmentation based on CNN and level set method. In Medical Imaging 2019: Image Processing, San Diego, California, United States, 16-21 February 2019, pp. 109492N. External Links: Link, Document Cited by: §4, §4.
  71. Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions. CoRR abs/1806.02051. External Links: Link, 1806.02051 Cited by: §6.4.
  72. Shape modeling with front propagation: A level set approach. IEEE Trans. Pattern Anal. Mach. Intell. 17 (2), pp. 158–175. External Links: Link, Document Cited by: §4.
  73. Deformable models in medical image analysis: a survey. Medical Image Anal. 1 (2), pp. 91–108. External Links: Link, Document Cited by: §1.1.
  74. Deep active shape model for robust object fitting. IEEE Trans. Image Process. 29, pp. 2380–2394. External Links: Link, Document Cited by: §3, §3.
  75. Conditional random fields incorporate convolutional neural networks for human eye sclera semantic segmentation. In 2017 IEEE International Joint Conference on Biometrics, IJCB 2017, Denver, CO, USA, October 1-4, 2017, pp. 768–773. External Links: Link, Document Cited by: §2, §2.
  76. Segmentation of magnetic resonance images using a combination of neural networks and active contour models. Medical engineering & physics 26, pp. 71–86. External Links: Document, Link Cited by: §4, §4.
  77. Incorporating prior shape knowledge via data-driven loss model to improve 3d liver segmentation in deep cnns. Int. J. Comput. Assist. Radiol. Surg. 15 (2), pp. 249–257. External Links: Link, Document Cited by: §1.1.
  78. Conditional random fields as recurrent neural networks for 3d medical imaging segmentation. CoRR abs/1807.07464. External Links: Link, 1807.07464 Cited by: §2, §2.
  79. A novel segmentation framework for uveal melanoma in magnetic resonance imaging based on class activation maps. In International Conference on Medical Imaging with Deep Learning, MIDL 2019, 8-10 July 2019, London, United Kingdom, M. J. Cardoso, A. Feragen, B. Glocker, E. Konukoglu, I. Oguz, G. B. Unal and T. Vercauteren (Eds.), Proceedings of Machine Learning Research, Vol. 102, pp. 370–379. External Links: Link Cited by: §3, §3.
  80. Ocular structures segmentation from multi-sequences MRI using 3d unet with fully connected crfs. In Computational Pathology and Ophthalmic Medical Image Analysis - First International Workshop, COMPAY 2018, and 5th International Workshop, OMIA 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16-20, 2018, Proceedings, D. Stoyanov, Z. Taylor, F. Ciompi, Y. Xu, A. L. Martel, L. Maier-Hein, N. M. Rajpoot, J. van der Laak, M. Veta, S. J. McKenna, D. R. J. Snead, E. Trucco, M. K. Garvin, X. J. Chen and H. Bogunovic (Eds.), Lecture Notes in Computer Science, Vol. 11039, pp. 167–175. External Links: Link, Document Cited by: §2, §2.
  81. Automatic lymph node cluster segmentation using holistically-nested neural networks and structured optimization in CT images. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016 - 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II, pp. 388–397. External Links: Link, Document Cited by: §2, §2.
  82. Incorporating prior knowledge in medical image segmentation: a survey. CoRR abs/1607.01092. External Links: Link, 1607.01092 Cited by: §1.1.
  83. Adaptive level set with region analysis via mask R-CNN: A comparison against classical methods. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, pp. 1–8. External Links: Link, Document Cited by: §4, §4.
  84. A survey of graph theoretical approaches to image segmentation. Pattern Recognit. 46 (3), pp. 1020–1038. External Links: Link, Document Cited by: §1.1.
  85. Persistent homology for fast tumor segmentation in whole slide histology images. Procedia Computer Science 90, pp. 119–124. Cited by: §5.
  86. A weakly supervised registration-based framework for prostate segmentation via the combination of statistical shape model and CNN. CoRR abs/2007.11726. External Links: Link, 2007.11726 Cited by: §3, §3.
  87. Inferring skin lesion segmentation with fully connected crfs based on multiple deep convolutional neural networks. IEEE Access 8, pp. 144246–144258. External Links: Link, Document Cited by: §2, §2.
  88. DeepCut: object segmentation from bounding box annotations using convolutional neural networks. IEEE Trans. Med. Imaging 36 (2), pp. 674–683. External Links: Link, Document Cited by: §2, §2.
  89. Deep learning for medical image processing: overview, challenges and future. CoRR abs/1704.06825. External Links: Link, 1704.06825 Cited by: §1.1.
  90. Neural persistence: a complexity measure for deep neural networks using algebraic topology. arXiv preprint arXiv:1812.09764. Cited by: §5.
  91. Deep learning approaches to biomedical image segmentation. Informatics in Medicine Unlocked 18, pp. 100297. External Links: ISSN 2352-9148, Document, Link Cited by: §1.1.
  92. Deep active contours. CoRR abs/1607.05074. External Links: Link, 1607.05074 Cited by: §4, §4.
  93. Fully automatic prostate segmentation in MR images using a new hybrid active contour-based approach. Signal, Image and Video Processing 12 (8), pp. 1629–1637. External Links: Link, Document Cited by: §4, §4.
  94. A method for semantic knee bone and cartilage segmentation with deep 3d shape fitting using data from the osteoarthritis initiative. In Shape in Medical Imaging - International Workshop, ShapeMI 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Proceedings, M. Reuter, C. Wachinger, H. Lombaert, B. Paniagua, O. Goksel and I. Rekik (Eds.), Lecture Notes in Computer Science, Vol. 12474, pp. 85–94. External Links: Link, Document Cited by: §3, §3.
  95. Sub-cortical brain structure segmentation using f-cnn’s. In 13th IEEE International Symposium on Biomedical Imaging, ISBI 2016, Prague, Czech Republic, April 13-16, 2016, pp. 269–272. External Links: Link, Document Cited by: §2, §2.
  96. Brain tumor segmentation using concurrent fully convolutional networks and conditional random fields. In Proceedings of the 3rd International Conference on Multimedia and Image Processing, ICMIP 2018, Guiyang, China, March 16-18, 2018, pp. 24–30. External Links: Link, Document Cited by: §2, §2.
  97. Fully connected crf with data-driven prior for multi-class brain tumor segmentation. pp. 1727–1731. External Links: Document, Link Cited by: §2, §2.
  98. Automatic kidney segmentation in 3d pediatric ultrasound images using deep neural networks and weighted fuzzy active shape model. In 15th IEEE International Symposium on Biomedical Imaging, ISBI 2018, Washington, DC, USA, April 4-7, 2018, pp. 1170–1173. External Links: Link, Document Cited by: §3, §3.
  99. Knee menisci segmentation using convolutional neural networks: data from the osteoarthritis initiative. Osteoarthritis and Cartilage 26, pp. . External Links: Document, Link Cited by: §3, §3.
  100. Deep semantic segmentation of natural and medical images: A review. CoRR abs/1910.07655. External Links: Link, 1910.07655 Cited by: §1.1.
  101. A deep level set method for image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support - Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, 2017, Proceedings, pp. 126–134. External Links: Link, Document Cited by: §4, §4.
  102. Shape constrained CNN for cardiac MR segmentation with simultaneous prediction of shape and pose parameters. CoRR abs/2010.08952. External Links: Link, 2010.08952 Cited by: §3, §3.
  103. DeepNAT: deep convolutional neural network for segmenting neuroanatomy. NeuroImage 170, pp. 434–445. External Links: Link, Document Cited by: §2, §2.
  104. A generic probabilistic active shape model for organ segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2009, 12th International Conference, London, UK, September 20-24, 2009, Proceedings, Part II, pp. 26–33. External Links: Link, Document Cited by: §3.
  105. Deep semantic segmentation of kidney and space-occupying lesion area based on SCNN and resnet models combined with sift-flow algorithm. J. Medical Systems 43 (1), pp. 2:1–2:12. External Links: Link, Document Cited by: §2, §2.
  106. Automatic left ventricle segmentation in short-axis MRI using deep convolutional neural networks and central-line guided level set approach. Comput. Biol. Medicine 122, pp. 103877. External Links: Link, Document Cited by: §4, §4.
  107. An automatic learning-based framework for robust nucleus segmentation. IEEE Trans. Med. Imaging 35 (2), pp. 550–566. External Links: Link, Document Cited by: §3, §3.
  108. Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images. Journal of Medical Imaging 6, pp. 1. External Links: Document, Link Cited by: §4, §4.
  109. Automatic bladder segmentation from CT images using deep CNN and 3d fully connected CRF-RNN. Int. J. Comput. Assist. Radiol. Surg. 13 (7), pp. 967–975. External Links: Link, Document Cited by: §2, §2.
  110. 3D fully convolutional network-based segmentation of lung nodules in CT images with a clinically inspired data synthesis method. In Medical Imaging 2019: Computer-Aided Diagnosis, San Diego, California, United States, 16-21 February 2019, pp. 109503G. External Links: Link, Document Cited by: §2, §2.
  111. Accurate and automatic tooth image segmentation model with deep convolutional neural networks and level set method. Neurocomputing 419, pp. 108 – 125. External Links: ISSN 0925-2312, Document, Link Cited by: §4, §4.
  112. An improved full convolutional network combined with conditional random fields for brain MR image segmentation algorithm and its 3d visualization analysis. J. Medical Systems 43 (9), pp. 292:1–292:10. External Links: Link, Document Cited by: §2, §2.
  113. Geometric loss for deep multiple sclerosis lesion segmentation. CoRR abs/2009.13755. External Links: Link, 2009.13755 Cited by: §1.1.
  114. Faster r-cnn, fourth-order partial differential equation and global-local active contour model (FPDE-GLACM) for plaque segmentation in IV-OCT image. Signal Image Video Process. 14 (3), pp. 509–517. External Links: Link, Document Cited by: §4, §4.
  115. Deep active contour network for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 - 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV, A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu and L. Joskowicz (Eds.), Lecture Notes in Computer Science, Vol. 12264, pp. 321–331. External Links: Link, Document Cited by: §4, §4.
  116. ARPM-net: A novel cnn-based adversarial method with markov random field enhancement for prostate and organs at risk segmentation in pelvic CT images. CoRR abs/2008.04488. External Links: Link, 2008.04488 Cited by: §2, §2.
  117. Improved nuclear segmentation on histopathology images using a combination of deep learning and active contour model. In Neural Information Processing - 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13-16, 2018, Proceedings, Part VI, pp. 307–317. External Links: Link, Document Cited by: §4, §4.
  118. Brain tumor segmentation using a fully convolutional neural network with conditional random fields. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries - Second International Workshop, BrainLes 2016, with the Challenges on BRATS, ISLES and mTOP 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 17, 2016, Revised Selected Papers, pp. 75–87. External Links: Link, Document Cited by: §2, §2, §2.
  119. A deep learning model integrating fcnns and crfs for brain tumor segmentation. Medical Image Analysis 43, pp. 98–111. External Links: Link, Document Cited by: §2, §2.
  120. Conditional random fields as recurrent neural networks. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1529–1537. External Links: Link, Document Cited by: §2.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
426419
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description