Convexity Shape Prior for Level Set based Image Segmentation Method
We propose a geometric convexity shape prior preservation method for variational level set based image segmentation methods. Our method is built upon the fact that the level set of a convex signed distanced function must be convex. This property enables us to transfer a complicated geometrical convexity prior into a simple inequality constraint on the function. An active set based Gauss-Seidel iteration is used to handle this constrained minimization problem to get an efficient algorithm. We apply our method to region and edge based level set segmentation models including Chan-Vese (CV) model with guarantee that the segmented region will be convex. Experimental results show the effectiveness and quality of the proposed model and algorithm.
Keywords: Convexity shape prior, Image segmentation, Level set method, Chan-Vese model.
Shape prior plays an important role in image segmentation, in which one of the most important is the convexity shape prior. A convexity shape or called a convexity region means that the points inside the region form a convex set. Things with convexity shape are quite common in our daily lives, such as a road sign, a book, or a basketball, etc.. Here, we propose a general convexity shape prior preservation method based on level set method. This method can be extended to many level set based image segmentation methods to get some convexity segmentation results.
Shape prior is widely studied in image segmentation. Leventon et al.  used statistical methods in shape prior methods. Rousson and Paragios  first applied level set method in shape prior methods. Chan and Zhu  purposed a theory which allows the shape prior to translate, scale and rotate based on level set method. Thiruvenkadam et al.  could segment multi objects from one single image, even the objects are duplicated or covered. Vu and Manjunath  combined shape prior methods with graph-cut method [6, 7]. Guo et al.  built a model which allows to choose a best shape prior from multi shape priors which are given by the user. The above works are all about some special shape priors, but in the nature, one thing may have several shapes, i.e., a general shape, such as convexity shape prior.
As for convexity shape prior, Liu et al.  studied the importance of convexity shape prior in image, as well as more studies in [10, 11, 12, 13]. The key of handling convexity shape prior is to describe convexity. Gorelick et al.  introduced some constraints on all the straight lines in the discrete image plane under graph theory, which ensures such a property: for any three points that lie on a straight line, if the outer two points were labeled as the object, then the middle point must also be labeled as the object, which is equal to the definition of convexity shape. Royer et al.  based on graph theory, defined the path and the straight line path between object points, to ensure that all the points which are on the straight path between two object points must be labeled with object. This property could also lead to a convexity shape. These two methods are based on graph theory which studied the connections between vertexes. However, such a graph based discrete methods could not be easily applied to variational PDE method since the geometrical constraints cannot be easily addressed in variational methods.
On variational PDE method, existing methods on convexity shape prior include [12, 13]. Strekalovskiy and Cremers  introduced an energy functional, which on each point defines the penalty of changing label in each direction. With the help of N auxiliary sub-region, this model could successfully segment a N-polygon. For example, if we want to get a circle as a segmentation result, N would be quite large, which requires many auxiliary sub-regions. Bae et al. used a relation between convexity and curvature in . They transferred convexity into a constraint on curvature, and proposed an Euler’s elastic based segmentation model. To optimize this problem, one needs to solve a nonlinear fourth order PDE. It would be time-consuming, although some recently developed operator splitting methods can be employed. Besides, it is not easy to strictly keep the constraint numerically, and thus some of the control parameters would play a very import role on segmentation results.
In this paper, we would like to propose a simple linear constraint for convexity shape prior. It is derived from an import property of a smooth signed distance function (SDF).
SDF is used in the reinitialization step of the level set function (LSF) evolution based on level set method (LSM) . [15, 16, 17, 18] proposed active contour based image segmentation models which took advantage of LSM. Chan and Vese  combine LSM with region based image segmentation method. Rada and Chen  used two level sets to achieve selective image segmentation. Compared with , Rada and Chen  achieved the same purpose using only one level set. Ali et al.  can segment multi-objects by only one level set. Although LSM is widely used, the LSF may develop irregularities during the level set evolution. Usually, a reinitialization of LSF is needed  during the curve evolutions. Exactly finding a SDF equals to solve the well-known Eikonal equation. For high computational efficiency, fast matching method  which is similar to Dijkstra’s algorithm  in graph can always be used to reinitialize SDF. Fast sweeping method  which uses Gauss-Seidel iteration with alternating sweeping ordering method is another alternative method for reinitialization task. To avoid direct reinitialization, Li et al.  proposed a penalty method by adding a nonconvex functional in the segmentation energy for the constraint . In papers [27, 28, 29], they extended this penalty method and proposed a two layers level set approach, the segmentation results are very similar to convexity shape. However, they could not give a theoretical condition to ensure convexity. In our numerical experiments, we will compare our new segmentation model with these edge and region based level set methods including the well-known Chan-Vese model.
In this paper, we show that the convex shape prior can be guaranteed by a simple constraint on the signed distance function. This constraint can be easily incorporated into variational models with LSM and solved efficiently by some constrained minimization techniques.
The main contributions of this paper are very clear: firstly, we propose a simple linear inequality constraint for convex shapes prior based on signed distance functions. Secondly, an efficient active set based Gauss-Seidel algorithm is proposed to numerically handle this inequality constraint. Numerical experiments on real and synthetic images are supplied to show the effectiveness and quality of the proposed model and algorithm.
The rest of our paper is organized as follows: firstly, in section 2, some related works on level set segmentation and convex shapes prior are introduced. Nextly, a sufficient condition for convex shape prior together with some mathematical analysis are given in section 3. In the following section 4, we apply our method on the level set based models. Section 5 contains an algorithm that can strictly keep the proposed inequality constraint. Experimental results and some comparisons are shown in section 7. Finally, we conclude this paper in section 8.
2 Related works
In this section, firstly, we will introduce the popular Chan-Vese model. Then secondly, some convex shape prior methods would be given.
2.1 Chan-Vese Model
Chan-Vese model is proposed by Chan and Vese in , which is an image segmentation model based on level set method and energy minimization method.
The first part of the energy functional is called data term
Here is an arbitrary closed curve, denotes the region inside curve , as well as denotes the region outside curve , and are unknown constants, which denote the inner and outer means of the two region and , respectively.
The second part of the energy functional is called regularization term, as
where denotes the Length of curve , denotes the area of region .
While in , we have the following equation,
Thus, we can set .
By introducing the Heaviside function , Dirac function and level set function , then (3) can be reformulated as
In implementation, and can be replaced by some smoothed version, see . It is easy to change the length term in the Chan-Vese model by a weighted length associated with an edge detector. Let us use the edge detector
Here G is a Gaussian kernel function, represents the convolution operator, and represents the gradient operator. The CV model with an edge detector is then
2.2 Convexity shape prior method by Gorelick et al.
Gorelick et al.  introduced a convexity constraint based on graph theory. In their work, the convexity of objects can be represented as a sum of 3-clique potentials by penalizing any 1-0-1 configuration on all straight lines. Here the label 1 denotes the object, the label 0 denotes the background in an image, with details below.
Define a function on all triplets of ordered pixels along all line , presents all the line on the image domain, as :
where denote the label of point , respectively, and denotes that the label of point is object, denotes that the label of point is background.
Then, they define the convexity energy as
It is easy to see that the target region is convex if and only if . By adding this energy to some proper chosen data term , where are the parameters estimated from a bounding box given by the users. Here is the Gibbs prior distribution. Thus the total energy becomes
Though this method can produce some approximate convex shape segmentations, it is not easy to extend it to general variational models for image segmentation. In the following, we focus on transferring the geometrical relationship to functions relationships.
3 Proposed Convex Shape Constraint
We will give some results with the help of implicit function presentation of a curve. Our main result of convex shape prior is built upon the signed distanced function.
Definition 3.1 (signed distance function, SDF ).
Let be a subset of with metric and be its boundary. A function : is called a signed distance function if:
where is the image domain.
In this paper, the metric is the commonly used Euclidean distance. To get a simple sufficient condition for convex region which is represented by SDF. In the following analyses, we assume that the SDF .
In geometry, convex set can be defined as
Definition 3.2 (Convex region).
is a convex region if and only if:
Using these two definitions and a mild smoothness assumption, we have some connections between signed distance function and convex region.
Let be a signed distance function of a region . If a.e. in and
Then we have that must be a convex region.
Before we give the proof of Theorem 3.3, let us give some useful propositions.
A region is a convex set if the SDF of is a convex function.
. Based on the definition of SDF,
Because is a convex function, then
Due to the definition of SDF,
For , is convex if the Hessian matrix of is positive semidefinite.
Let us denote , then
For , the Hessian matrix of is positive semidefinite if
[Basic Property of SDF] For any SDF , its gradient satisfies the eikonal equation: .
If a SDF a.e. in , then one can get a.e. in .
From Proposition 3.7, we have
Differential by and separately, we have
Transfer the second term to the right side, and multiply this two equation, we have
If , we have .
If or , then or , still holds.
Combine all above, we have a.e.. ∎
Now we can prove the theorem 3.3.
Using the definition of SDF and the proposition 3.8, we know that
Combining with the assumption , and apply the proposition 3.6, we know that the Hessian matrix of is positive semidefinite at each point. By Proposition 3.5, we get is a convex function. Therefore, we get is a convex region according to proposition 3.4. ∎
In the next, we shall propose some proper algorithms for variational level set methods with these constraints to guarantee convexity shape prior. The new models would be very simple but very efficient to keep convex shape prior.
4 Apply to the proposed convexity shape prior method on Chan-Vese model
Then the Chan-Vese model with convexity shape prior would be
The only difference between the original Chan-Vese model and (6) is that there is a constraint in (6), and it can guarantee that must be a convex function. Then the segmented result must be a convex region according to theorem 3.3.
5 Numerical techniques for the constrained minimization problems
Here, we build an algorithm for finding convex shape prior . For such a constraint, penalty method, Lagrange multiplier method and augmented Lagrange multiplier methods  etc. can be applied. In this work, we would not apply these methods since they are relaxation methods. Namely, the constraint can not be strictly kept during the iterations. In fact, the constraint should be strictly kept at each step in order to fulfill the convexity shape prior in our method. We have tested the penalty and Lagrange multiplier methods, they failed for our method in real implementations for holding the constraints. The constraint set is a convex set, and one possible optimiziation technique to to handle this constraints for minimize (6) is to use a projection algorithm. However, such a projection does not have a closed-form solution and we have to propose an algorithm to fullfil this convex constraint almost everywhere in .
For , we would like to split this problem into two subproblems which involve the constraints and seperately.
This constraint is non-convex. Since is very important for our method because of proposition (3.8). Penalty method such as  can not be used since we need to keep the condition almost everywhere and numerically this constraint must be kept with good accuracy at each grid point. Thus we propose to use the fast matching method  to reinitialize the level set function. It can be easily implemented by a Matlab function “bwdist”. For better expression, we use to denote this pseudo projection, i.e. is an optimal approximation of in the set . We summarize this method in Algorithm 1.
It is quite easy to verify that the set is a convex closed set in , and the Laplace operator is a linear operator. For a given function , we shall consider the projection onto , namely
By duality, this constrainted minimization problem have the following dual energy
Assuming is a saddle point of , then following the well-known KKT conditions, we have
Unfortunately, the closed-form solutions of and are not available since there is a second order differentiable operator . However, we can use (7-10) to build an iteration method to find an approximation of and using active set method.
For the -th iteration, denote the inactive set as for a given . The active set is . On the -th iteration, we want to find a to strictly satisfy all the KKT conditions. First, we calculate the corresponding inactive set , and we get for according to (9). Then we can set to satisfy (7). Otherwise, if , the constraint is active and we can solve the equation to satisfies (8). Here, we can use Gauss-Seidel iteration to get . Namely, . Here means the 4 neighborhood of point .
Now under a fixed active set , we have an approximate solution of our KKT condition. Since the active set is only available if we have the real minimal, here we have to use an iteration method to find a solution that close to the real minimal. In the whole iteration, we set iteration number of the Gauss-Seidel method just to be one. If inactive set and both converge, we get the final result , where .
We summary the Algorithm for calculating in Algorithm 2.
Input: , output: .
5.3 Summary of finding
Here, we summarize the algorithms for finding , denote as .
6 Numerical Algorithms for Chan-Vese model and edge based model with convex shape prior
The following alternating minimization algorithm which has often be used in the literature for the Chan-Vese model can be used to solve the Chan-Vese convex shape prior minimization problem (6):
Here the operator is given by Algorithm 3.
In our experiments, we use for (5). However, for some natural images, the foreground and background are not homogeneous and thus we will only use the edge force to segment these images. For such cases, we will take in (5) and refer to this as the edge based model. Correspondingly, there is no need to compute and update the constants when edge based model is used.
7 experiment results
Now, we demonstrate the segmentation results in this section. The computing platform for our numerical experiments is a laptop equipped with Intel Core i7 CPU @ 2.90GHz processor and 8G memory. The codes are written in Matlab R2013a without any special optimizing.
The Gaussian kernel used in our experiments is with size and the standard deviation . For the color images, we extract the biggest contrast channel of the RGB color space, and treat them as gray images for segmentation.
7.1 Testing of Algorithm 3
Firstly we test our Algorithm 3 on a synthetic non-convex region. We will show how the algorithm changes a non-convex region into a convex region. The size of test image is .
The test results are shown in Figure 1, where the images on the first row is the non-convex region and the results of iteration 4, 25, 150, respectively. Here this iteration number refers to the number in Algorithm 3, the max iteration of m in Algorithm 2 is set to 30. The second row is the corresponding level set function in green, with the zero level set in blue.
To show that our method successfully keeps the constraint, we show the values of the Laplace of the level set function in the third row. To be more clearly, we set the range of these values into [-0.2,0.2], as all we concern about is the relation with 0. For display, two value control dots are added in the upper left and down right corner. We use color blue to denote the positive value and red to denote the negative value. One can see from the images that the red zone becomes less and less and disappear at convergence. This means the Laplacian of the level set function is non-negative.
As you can see that the non-convex region is growing into a convex region near its convex hull. Near the part of the contour where the non-convex gap is wide (the right side), it needs more iterations to become convex. For the part of the contour where the gap is thin (the middle one), it turns to be convex in a few iteraions. This experiment shows that our method successfully turns a non-convex region into a convex region through its SDF using variational methods.
7.2 Comparisons between Chan-Vese model, proposed convex prior model (region based), edge based model, proposed convex prior model (edge based) and  on synthetic images
In this section we apply these segmentation methods to the same images and compare the segmentation results. The images are taken from MPEG-7 CE-Shape-1 Dataset https://cis.temple.edu/~latecki/. The sizes of image are . The parameters we used in this experiment are as follows: for the Chan-Vese model, , as well as the proposed convex prior model (region based).
Comparison results are listed in Figure 2, where the first column contains the original images, the second column to fifth column is the segmentation results of method Chan-Vese, proposed convex prior model (region based), edge based, proposed convex prior model (edge based), respectively. For better comparison, the manually label seeds of  is given in the sixth column. Here the blue color represents object, and the red represents background. The seventh column is the results of . As you can see that our algorithm successfully segments the region as a convex region in the third and fifth columns. Compared to the semi-supervised method  with discrete method, the results produced by the proposed are more smooth and tend to be more convex. In addition, our algorithm does not need manually labeled seeds.
7.3 Comparisons between models with and without convexity shape prior on real images
Here, we compare the models with convexity shape prior and without convexity shape prior, of both Chan-Vese model and edge based model, on some nature images. The image is a box with some leafs on it, the size of the image is , parameters appeared in the model are set as follows: for Chan-Vese model, . Note that our convexity shape prior is quite successful.
In Figure 3, the first row left is the original image, the right is the image with initial curve with yellow color for all the experiments. The second row is the segmentation result of Chan-Vese model, proposed convex prior model (region based), edge based model, proposed convex prior model (edge based), respectively. The third and fourth row has the same setting as the third row in Figure 1.
7.4 Initialization effects
The solution found by the proposed convex prior model with the proposed algorithm might not be a global minimizer, but it must be a convexity region. So initialization is important for our algorithm. Different initializations may lead to different segmentation results. To be more comparable, we also add the segmentation results of Chan-Vese model.
This phenomena is shown and compared in Figure 4. The first column is the original image with the initialization of each method, presented as a green curve. In the following, second to fifth columns, we show the segmentation results of Chan-Vese model, proposed convex prior model (region based), edge based model, proposed convex prior model (edge based), respectively, the segmentation contours are displayed by cyan color. The size of image is .
7.5 Results on some natural images
To further test our algorithms, we apply the proposed convex prior model (edge based) on more real images in Figure 5, with the green curves as initial curves, the red curves as the segmentation results. The image was resized to .
7.6 Test on images with occlusions
In this part, we apply the proposed convex prior model (edge based) on a real image in Figure 6. This image is a road sign, where the right down corner was covered by branches of a tree. The original image was from internet https://seeclickfix.com/issues/1996027-abq-ride-please-move-bus-stop-sign-post-blocked-by-tree, we are using part of it. The image was resized to .
In Figure 6, (a) is the original image, (b) is the original image with initial curve, (c) is the segmentation result without the convex prior, (d) is the segmentation result of proposed model. As you can see, our model successfully segment the whole road sign.
8 Conclusion and Discussion
In this paper, we proposed a convexity shape prior method for level set based segmentation method such as Chan-Vese model and edge based model. Through analyzing the connection of convex region and the corresponding signed distance function, we give out our convexity shape prior. By adding such a convex shape prior constraint into these models, we get image segmentation models with convexity shape prior. Compared with the non-convex segmentation, we can see that our method successfully segmented the desired region as a convex region. Although the algorithm works well and is very efficient, we do not yet know any convergence results for the proposed algorithm. In addition, the proposed method can not be directly extended to some region characteristic functions based segmentation models such as K-means, C-fuzzy, Potts model and GMM based EM segmentations. This is because the convex property in theorem 3.3 would be lost in such methods without signed distance functions. In the future, we may work on these aspects and extend our method to other applications such as medical images segmentations.
Shi Yan was partly supported by the China Scholarship Council and University of Bergen. Jun Liu and Haiyang Huang were partly supported by The National Key Research and Development Program of China (2017YFA0604903). The authors would like to thank for the DRLSE codes provided by Chunming Li from website: http://www.imagecomputing.org/~cmli/DRLSE/, and the codes provided by Lena Gorelick on her page http://www.csd.uwo.ca/~ygorelic/publications.html.
-  M.E. Leventon, W.E.L. Grimson, O. Faugeras. Statistical shape influence in geodesic active contours[C]. Proceedings of Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, volume 1. IEEE, 2000. 316–323.
-  M. Rousson, N. Paragios. Shape priors for level set representations[C]. Proceedings of Computer Vision, European Conference on. Springer, 2002. 78–92.
-  T.F. Chan, W. Zhu. Level set based shape prior segmentation[C]. Proceedings of Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 2. IEEE, 2005. 1164–1170.
-  S.R. Thiruvenkadam, T.F. Chan, B.W. Hong. Segmentation under occlusions using selective shape prior[C]. Proceedings of International Conference on Scale Space and Variational Methods in Computer Vision. Springer, 2007. 191–202.
-  N. Vu, B.S. Manjunath. Shape prior segmentation of multiple objects with graph cuts[C]. Proceedings of Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008. 1–8.
-  Y. Boykov, O. Veksler, R. Zabih. Fast approximate energy minimization via graph cuts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11):1222–1239.
-  V. Kolmogorov, R. Zabin. What energy functions can be minimized via graph cuts?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(2):147–159.
-  W. Guo, J. Qin, S. Tari. Automatic prior shape selection for image segmentation[C]. Proceedings of Research in Shape Modeling. Springer, 2015: 1–8.
-  Z. Liu, D.W. Jacobs, R.Basri. The role of convexity in perceptual completion: Beyond good continuation[J]. Vision Research, 1999, 39(25):4244–4257.
-  L. Gorelick, O. Veksler, Y. Boykov, et al. Convexity Shape Prior for Binary Segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(2):258–271.
-  L.A. Royer, D.L. Richmond, C. Rother, et al. Convexity Shape Constraints for Image Segmentation[C]. Proceedings of Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. 402–410.
-  E. Strekalovskiy, D. Cremers. Generalized ordering constraints for multilabel optimization[C]. Proceedings of Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 2011. 2619–2626.
-  E. Bae, X.C. Tai, W. Zhu. Augmented lagrangian method for an eulerâs elastica based segmentation model that promotes convex contours[J]. Inverse Problems and Imaging, 2017, 11(1):1–23.
-  S. Osher, J.A. Sethian. Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations[J]. Journal of Computational Physics, 1988, 79(1):12–49.
-  M. Kass, A. Witkin, D. Terzopoulos. Snakes: Active contour models[J]. International Journal of Computer Vision, 1988, 1(4):321–331.
-  V. Caselles, F. Catté, T. Coll, et al. A geometric model for active contours in image processing[J]. Numerische Mathematik, 1993, 66(1):1–31.
-  V. Caselles, R. Kimmel, G. Sapiro. Geodesic active contours[J]. International Journal of Computer Vision, 1997, 22(1):61–79.
-  A.J. Yezzi, S. Kichenassamy, A. Kumar, et al. A geometric snake model for segmentation of medical imagery.[J]. IEEE Transactions on Medical Imaging, 1997, 16(2):199–209.
-  T.F. Chan, L.A. Vese. Active contours without edges[J]. IEEE Transactions on Image Processing, 2001, 10(2):266–277.
-  L. Rada, K. Chen. A New Variational Model with Dual Level Set Functions for Selective Segmentation[J]. Communications in Computational Physics, 2012, 12(1):261–283.
-  L. Rada, K. Chen. Improved selective segmentation model using one level-set[J]. Journal of Algorithms and Computational Technology, 2013, 7(4):509–540.
-  H. Ali, N. Badshah, K. Chen, et al. Multiphase segmentation based on new signed pressure force functions and one 1 level set function[J]. Turkish Journal of Electrical Engineering and Computer Sciences, 2017, 25(4):2943–2955.
-  J.A. Sethian. Level set methods and fast marching methods: evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science[M]. Cambridge University Press, 1999.
-  E. W. Dijkstra. A note on two problems in connexion with graphs[J]. Numerische Mathematik, 1959, 1(1):269–271.
-  H.K. Zhao. A fast sweeping method for Eikonal equations[J]. Mathematics of Computation, 2005, 74(250):603–627.
-  C. Li, C. Xu, C. Gui, et al. Distance regularized level set evolution and its application to image segmentation[J]. IEEE Transactions on Image Processing, 2010, 19(12):3243–3254.
-  Y. Liu, C. Li, S. Guo, et al. A novel level set method for segmentation of left and right ventricles from cardiac MR images[C]. Proceedings of 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2014. 4719–4722.
-  C. Feng, C. Li, D. Zhao, et al. Segmentation of the Left Ventricle Using Distance Regularized Two-Layer Level Set Approach[C]. In: Kensaku Mori, Ichiro Sakuma, Yoshinobu Sato, et al., (eds.). Proceedings of Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013, Berlin, Heidelberg: Springer Berlin Heidelberg, 2013. 477–484.
-  C. Feng, S. Zhang, D. Zhao, et al. Simultaneous extraction of endocardial and epicardial contours of the left ventricle by distance regularized level sets[J]. Medical Physics, 43(6Part1):2741–2755.
-  V. Estellers, D. Zosso, R. Lai, et al. Efficient algorithm for level set method preserving distance function[J]. IEEE Transactions on Image Processing, 2012, 21(12):4722–4734.
-  C. Wu, X.C. Tai. Augmented Lagrangian method, dual methods, and split Bregman iteration for ROF, vectorial TV, and high order models[J]. SIAM Journal on Imaging Sciences, 2010, 3(3):300–339.