Multidimensional TV-Stokes for image processing

Multidimensional TV-Stokes for image processing

Abstract

A complete multidimential TV-Stokes model is proposed based on smoothing a gradient field in the first step and reconstruction of the multidimensional image from the gradient field. It is the correct extension of the original two dimensional TV-Stokes to multidimensions. Numerical algorithm using the Chambolle’s semi-implicit dual formula is proposed. Numerical results applied to denoising 3D images and movies are presented. They show excellent performance in avoiding the staircase effect, and preserving fine structures.

T
\epstopdfDeclareGraphicsRule

.tifpng.pngconvert #1 \OutputFile \AppendGraphicsExtensions.tif \apptocmd \headersMultidimensional TV-Stokes for image processingB. Wu, X. C. Tai, and T. Rahman

otal variation, Denoising, Restoration

1 Introduction

As one of the major techniques in computer vision, image processing is widely used in the community. One of the important tasks of image processing is image denoising which is used for removing noises from corrupted images. It plays a crucial and distinguished role, e.g. being used as a pre-step to any processing or analysis of the images. The noise in a corrupted image may depend on the type of instrument used, the procedure applied, and the environment in which the image is taken. Noises are characterized as Gaussian or salt-pepper, additive or multiplicative [3]. We consider only the Gaussian noise in this paper, which is additive.

Searching for a more general and robust algorithm to handle the problem of image denoising has been one of the major interests in the image processing community. With the pioneering work of Rudin, Osher and Fatemi (cf. [18]) based on the total variation minimization for image denoising, published in 1992, their model has been widely used as the basis for most denoising algorithms because of its elegant mathematics, simplicity of implementation, and effectiveness. Allthough the model is able to preserve edges in the image, which is important for an image, it has the limitation that it exhibits staircase effect or produces patchy images, cf. [16]. A number of algorithms now exist that are based on the total variation minimization, which try to overcome this limitation, including high order algorithms, cf. [13, 21], the training based algorithms, cf. [5], iterative regularization algorithms, cf. [15, 4, 14], and algorithms based on the TV-Stokes, cf. [20, 16, 12, 9, 7, 8]. There are also excellent algorithms that are not based on the variational approach, e.g., the non-local algorithms, cf. [1], the block matching algorithm, cf. [6], and the low rank algorithm, cf. [11].

The purpose of this paper is to propose a model which can be used for the data of any dimensions, e.g. 3D medical data, 2D video, and dynamic volume data. The model is expected to be simple but effective with only a small number of parameters. The TV-Stokes model is the right candidate for this. We extend the TV-Stokes model from 2D to an arbitrary dimensions, and we call it the multi-dimensional TV-Stokes. Even though the paper addresses the task of denoising, The model can be extended for use in other data restoration tasks, like image inpainting, image compression, etc. .

The structure of the paper is organized as follows. In Section 2, we present the first step of the multi-dimensional TV-Stokes and its dual formulation. is addressed accompanying with related analysis. In the Section 3, Chambolle’s semi-implicit iteration is presented for the minimization. The key part of the multi-dimensional TV-Stokes is the orthogonal projection onto the constrained spaces, which is presented in Section 4, based on the singular value decomposition of the finite difference matrix . Section 5 gives the second step of the proposed model, height map reconstruction using a surface fitting technique. Numerical results are presented in Section 6, comparing the proposed model with the Rudin-Osher-Fatemi (ROF) model.

2 The vector field smoothing step of the TV-Stokes

The first step of the TV-Stokes is the vector field smoothing step. Its required that the desired smoothed vector field is a gradient field, that is where and is a -dimensional grayscale image. In the case , it is the same as requiring for the gradient field, which is equivalent to , which is exactly the constraint used in the TV-Stokes model for 2D.

Let denote the components of a -dimensional grayscale image with components for . When a linear transformation defined by a matrix is applied to the -th dimension of , its result will be denoted by . For instance,

The gradient field of is , where signifies a discrete gradient operator with homogeneous Neumann boundary conditions. Namely, the gradient operator is determined by means of the following differentiation matrix

(1)

Given a -dimensional image corrupted by a Gaussian noise, a constrained variant of the Rudin-Osher-Fatemi model [18] consists in finding a gradient field satisfying the minimization problem

(2)

where is a suitable scalar parameter, and

The (isotropic) norms in (2) are as follows:

The linear subspace is the range of the orthogonal projector

(3)

where is the adjoint to the operator, and denotes the Moore-Penrose pseudoinverse. By the way, .

The continuous functional

is strictly convex, its minimum is unique and attained in the closed ball .

By the aid of a dual variable

such that the model (2) reduces to the equivalent min-max formulation

(4)

Here the max norm is .

The generalized minimax theorem, cf. [19] , justifies the equality

so that the model (2) reduces to the max-min formulation

(5)

In spite of a possible nonuniqueness of , the solution obtained by means of (5) is unique. Indeed, if and are solutions of (4) and (5) respectively, then by Lemma 36.2 in [17]. The strict convexity of the mapping implies .

Since the constraint is equivalent to the condition , the term in (5) can be replaced by . Now we observe that the unique solution

(6)

of the unconstrained problem also solves the constrained problem .

Under the condition (6)

and we arrive at the minimum distance problem

(7)

2.1 Chambolle’s iteration

We solve the minimization problem (7) using Chambolle’s iteration, cf. [2, 7]. Accordingly, the Karush-Kuhn-Tucker conditions for (7) are given by the equation

(8)

where the dot signifies the entrywise product. Solution of (7) is approximated by the semi-implicit iteration

(9)

where the maximum and division are both entrywise operations.

Lemma 1

The semi-implicit iteration (9) is 1-Lipschitz [10] for

(10)

Proof

Each step of (9) uses two mappings: and . The first mapping is linear and 1-Lipschitz if and only if , where is the identity transformation. The second mapping is always 1-Lipschitz.

The SVD of the matrix allows us to show that 1-Lipschitz holds when

(11)

2.2 Implementation of

The intricate part in our algorithm is the implementation of the operation. We describe here how this can be done efficiently. The singular value decomposition of the differentiation matrix is

where the diagonal has the entries , . The orthogonal matrix of the discrete cosine transform has the entries

The orthogonal matrix of the discrete sine transform has the entries

We can use the singular value decomposition of to compute . Indeed, .

We implement by the fast Fourier transform. Since the equation is equivalent to

where and . The components of and are related by the equalities

Note that equals zero if and is positive otherwise. Hence

Recall that . Moreover, multiplication with the matrices and should be carried out by the fast Fourier transform FFT.

3 The scalar field reconstruction step of the TV-Stokes

We extend the second step or the image reconstruction step of the original TV-Stokes [16] for 2D to multidimensions. Accordingly, for the given -dimensional image , which is corrupted by a Gaussian noise, a vector matching model [16] is applied which satisfies the minimization problem,

(12)

where is a suitable scalar parameter, and

Note that the is obtained from the first step and thus, in the reconstruction, it is a known term. By completing the quadratic term, the above problem (12) can be reformed as

(13)

which is strictly convex, its minimum is unique and attained in the closed ball .

With the aid of a dual variable such that the model (12) reduces to the equivalent min-max formulation

(14)

Here the max norm is .

The generalized minimax theorem, cf. [19], justifies the equality

so that the model (12) reduces to the max-min formulation

(15)

In spite of a possible nonuniqueness of , the solution obtained by means of (15) is unique. Indeed, if and are solutions of (14) and (15) respectively, then by (Lemma 36.2 of [17]). The strict convexity of the mapping implies . The unique solution is obtained as follows

(16)

Under the condition (16)

and we arrive at the minimum distance problem

(17)

As in [2, 7] the Karush-Kuhn-Tucker conditions for (17) are given by the equation

(18)

where the dot signifies the entrywise product. Solution of (17) is approximated by the semi-implicit iteration

(19)

where the maximum and division are entrywise operations.

Lemma 2

The semi-implicit iteration (19) is 1-Lipschitz for

(20)

Proof

Each step of (19) uses two mappings:

and . The first mapping is linear and 1-Lipschitz if and only if , where is the identity transformation. The second mapping is always 1-Lipschitz.

The SVD of the matrix allows us to show that 1-Lipschitz holds when

(21)

4 Numerical experiments

We first test the proposed model on some 3d static data, which are computer tomography data from two lung tissues of human. In Fig. 1, Fig. 3 and Fig. 4, 3 groups of orthogonal slices are presented, with respect to -plane, -plane, and -plane. In these figures, the last rows show relative positions of those slices in their embedded volume data. In the first column of each figures, the 3-slice view is taken on the raw data, which contains some unknown noise. The noise type is supposed to be Gaussian. In the middle column and the last column of each figures, comparison is presented between the denoised data of using ROF model and the proposed multidimensional TV-Stokes model. The one (last column) using the proposed multidimensional TV-Stokes model results higher fidelity to the raw data than the one (middle column) using ROF model and preserves more fine structures as well as better continuity of tube-like structures. Fig. 2 and Fig. 5 show the isosurfaces of the volume data. In these figures, the results from proposed model (last column) well trade-off the smoothness and details preservation. It observes that the fine tube-like structures disappeared while the ROF model smooths the noisy data.

(a) (b) (c) (d) (e) (f) (g) (h) (i)
Figure 1: Visualization of computer tomography data of human lung tissue A via 3 groups of orthogonal slices. Row one, row two and row three present respectively the -slices, the -slices and the -slices. Column (a) visualizes the raw data with some noise (Gaussian noise). Column (b) visualizes the data denoised with ROF model. Column (c) visualizes the data denoised with proposed multidimensional TV-Stokes model.
(a) (b) (c)
Figure 2: Isosurface show of computer tomography data of human lung tissue A. (a) raw data with some noise (Gaussian noise). (b) data denoised with ROF model. (c) data denoised with proposed multidimensional TV-Stokes model.
(a) (b) (c) (d) (e) (f) (g) (h) (i)
Figure 3: Visualization of computer tomography data of human lung tissue B via 3 groups of orthogonal slices. Row one, row two and row three present respectively the -slices, the -slices and the -slices. Column (a) visualizes the raw data with some noise (Gaussian noise). Column (b) visualizes the data denoised with ROF model. Column (c) visualizes the data denoised with proposed multidimensional TV-Stokes model.
(a) (b) (c) (d) (e) (f) (g) (h) (i)
Figure 4: Visualization of computer tomography data of human lung tissue B via 3 groups of orthogonal slices. In contrast to Fig. 3, different observation slices are presented. Row one, row two and row three present respectively the -slices, the -slices and the -slices. Column (a) visualizes the raw data with some noise (Gaussian noise). Column (b) visualizes the data denoised with ROF model. Column (c) visualizes the data denoised with proposed multidimensional TV-Stokes model.
(a) (b) (c) (d) (e) (f)
Figure 5: Isosurface show of computer tomography data of human lung tissue B. (a) raw data with some noise (Gaussian noise). (b) data denoised with ROF model. (c) data denoised with proposed multidimensional TV-Stokes model. In row two, only tube-like structures are presented.

Another experiment is performed on 2d dynamic data, where new 3d data are obtained by stacking all 2d data along temporal dimension. Fig. 6 and Fig. 7 show the results of using proposed multidimensional TV-Stokes on movie / monitoring video denoising. In Fig. 6, smooth cheek and sharp edges are well balanced, showing in row two and row four. In escalator case, cf. Fig. 7, shape edges of structures and digitals are preserved, showing in row two and row four.

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) (q) (r) (s) (t) (u) (v) (w) (x)
Figure 6: Contrast of initial noisy movie and its denoised copy, by using proposed multidimensional TV-Stokes model. Row one and row three are sequences from initial movie. Row two and row four are sequences from denoised results.
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o) (p) (q) (r) (s) (t)
Figure 7: Contrast of initial noisy monitoring video and its denoised copy, by using proposed multidimensional TV-Stokes model. Row one and row three are sequences from initial video. Row two and row four are sequences from denoised results.

Acknowledgments

We thank Dr. Andreas Langer for interesting discussion and for providing us with the 3d data.

Footnotes

  1. footnotemark:

References

  1. A. Buades, B. Coll and J. Morel (2005) A Non-Local Algorithm for Image Denoising. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 60–65. External Links: ISBN 0-7695-2372-2, ISSN 1063-6919 Cited by: §1.
  2. A. Chambolle (2004) An Algorithm for Total Variation Minimization and Applications. Journal of Mathematical Imaging and Vision 20 (2), pp. 89–97. Cited by: §2.1, §3.
  3. T. F. Chan and J. Shen (2005) Image processing and analysis: variational, pde, wavelet, and stochastic methods. SIAM. Cited by: §1.
  4. M. R. Charest, M. Elad and P. Milanfar (2006) A general iterative regularization framework for image denoising. 2006 IEEE Conference on Information Sciences and Systems, CISS 2006 - Proceedings 1 (3), pp. 452–457. External Links: ISBN 1424403502 Cited by: §1.
  5. Y. Chen and T. Pock (2017) Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (6), pp. 1256–1272. External Links: 1508.02848, ISSN 01628828 Cited by: §1.
  6. K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian (2007) Image Denoising by Sparse 3D Transformation-Domain Collaborative Filtering. IEEE Transactions on Image Processing 16 (8), pp. 1–16. External Links: arXiv:1011.1669v3, ISBN 1057-7149 VO - 16, ISSN 1057-7149, Link Cited by: §1.
  7. C. A. Elo, A. Malyshev and T. Rahman (2009) A Dual Formulation of the TV-Stokes Algorithm for Image Denoising. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5567 LNCS (3), pp. 307–318. External Links: ISBN 3642022553, ISSN 03029743 Cited by: §1, §2.1, §3.
  8. C. A. Elo (2009) Image Denoising Algorithms Based on the Dual Formulation of Total Variation. Master thesis, University of Bergen. External Links: Link Cited by: §1.
  9. J. Hahn, C. Wu and X. Tai (2012) Augmented Lagrangian Method for Generalized TV-Stokes Model. Journal of Scientific Computing 50 (2), pp. 235–264. External Links: ISSN 0885-7474, Link Cited by: §1.
  10. J. Heinonen (2012) Lectures on analysis on metric spaces. Springer Science & Business Media. Cited by: Lemma 1.
  11. H. Ji, C. Liu, Z. Shen and Y. Xu (2010) Robust Video Denoising Using Low Rank Matrix Completion. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1791–1798. External Links: ISBN 978-1-4244-6984-0, ISSN 1063-6919, Link Cited by: §1.
  12. W. G. Litvinov, T. Rahman and X. Tai (2011) A Modified TV-Stokes Model for Image Processing. SIAM Journal on Scientific Computing 33 (4), pp. 1574–1597. External Links: ISSN 1064-8275 Cited by: §1.
  13. M. Lysaker and X. C. Tai (2006) Iterative Image Restoration Combining Total Variation Minimization and a Second-Order Functional. International Journal of Computer Vision 66 (1), pp. 5–18. External Links: ISSN 09205691 Cited by: §1.
  14. L. Marcinkowski and T. Rahman (2016) An Iterative Regularization Algorithm for the TV-Stokes in Image Processing. In Parallel Processing and Applied Mathematics, Lecture Notes in Computer Science, Vol. 9574, pp. 381–390. External Links: ISBN 978-3-319-32151-6 Cited by: §1.
  15. S. Osher, M. Burger, D. Goldfarb, J. Xu and W. Yin (2005) An Iterative Regularization Method for Total Variation-Based Image Restoration. Multiscale Modeling & Simulation 4 (2), pp. 460–489. External Links: ISBN 0898591910, ISSN 1540-3459 Cited by: §1.
  16. T. Rahman, X. Tai and S. Osher (2007) A TV-Stokes Denoising Algorithm. In Scale Space and Variational Methods in Computer Vision, Lecture Notes in Computer Science, pp. 473–483. External Links: ISBN 9783540728221 Cited by: §1, §3.
  17. R. Rockafellar (1997) Convex analysis. Princeton University Press, Princeton, NJ. External Links: ISBN 9780691015866 Cited by: §2, §3.
  18. L. I. Rudin, S. Osher and E. Fatemi (1992) Nonlinear Total Variation Noise Removal Algorithm. Physica D: Nonlinear Phenomena 60 (1-4), pp. 259–268. External Links: arXiv:1011.1669v3, ISBN 0167-2789, ISSN 01672789, Link Cited by: §1, §2.
  19. M. Sion (1957) On General Minimax Theorems. Pacific Journal of Mathematics 18 (4), pp. 171–176. External Links: ISSN 0030-8730 Cited by: §2, §3.
  20. X. Tai, S. Osher and R. Holm (2006) Image Inpainting Using a TV-Stokes Equation. CAM report 06 (01), pp. 1–20. External Links: Link Cited by: §1.
  21. B. Wu, T. Rahman and X. C. Tai (2018) Sparse-Data Based 3D Surface Reconstruction for Cartoon and Map. Mathematics and Visualization 0 (221219), pp. 47–64. External Links: ISBN 9783319912745, ISSN 2197666X Cited by: §1.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
414607
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description