Convolutional Dictionary Regularizers for Tomographic Inversion

Convolutional Dictionary Regularizers for Tomographic Inversion


There has been a growing interest in using data-driven regularizers to solve inverse problems associated with computational imaging systems. The convolutional sparse representation model has recently gained attention, driven by the development of fast algorithms for solving the dictionary learning and sparse coding problems for sufficiently large images and data sets. Nevertheless, this model has seen very limited application to tomographic reconstruction problems. In this paper, we present a model-based tomographic reconstruction algorithm using a learnt convolutional dictionary as a regularizer. The key contribution is the use of an iteration dependent weighting scheme to construct an effective denoising method that is integrated into the inversion using the Plug-and-Play reconstruction framework. Using simulated data sets we demonstrate how our approach can improve performance over traditional regularizers based on a Markov random field model and a patch-based sparse representation model model for sparse and limited-view tomographic data sets.


S. V. Venkatakrishnan thanks: This manuscript has been authored by UT-Battelle, LLC, under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan ( S.V. Venkatakrishnan and B.Wohlberg were supported via the Laboratory Directed Research and Development program at Oak Ridge National Lab and Los Alamos National Lab respectively.   Brendt Wohlberg \address Imaging, Signals and Machine Learning Group, Oak Ridge National Lab, Oak Ridge, TN 37831
Theoretical Division, Los Alamos National Laboratory, Los Alamos, NM 87545

1 Introduction

Model-based reconstruction algorithms have enabled dramatic improvements in the performance of tomographic reconstructions compared to traditional approaches, especially for sparse, limited-view and noisy data sets [1]. These methods solve the tomographic reconstruction by minimizing a cost function that balances a data-fidelity term and a regularization term that promotes certain desirable properties of the reconstruction itself. While model-based methods have helped to improve the performance of tomographic imaging systems, the potential to further improve the quality by using different regularizers is still being explored.

Several regularizers have been proposed to improve the quality of model-based tomographic reconstructions. These include the edge-preserving total-variation model [2], the non-local self-similarity model [3], and those that constrain the solution to a sparse combination of elements from an over-complete dictionary based on wavelets or other transforms. Data-driven regularizers that learn a model from an off-line database have also been applied to tomographic inversion [4, 5, 6]. Among data-driven regularizers, patch based dictionary models [4, 6, 7] have been widely developed for tomography, with promising performance. However, the patch is a local model and can result in redundant dictionary elements that are merely translated versions of each other. As a result there has been a revival of interest in the use of shift-invariant [8] models for images, also called convolutional sparse representation (CSR) models [9, 10]. Recent work on efficient algorithms for convolutional dictionary learning (CDL) [11, 12, 13, 14, 15] and the corresponding convolutional sparse coding (CSC) [16, 17] problem have allowed for the use of CSR as regularizers for a variety of inverse problems [18, 19, 20, 21].

Existing approaches to exploiting the CSR model for tomography and related problems [22, 23] have integrated the inversion into a CDL problem, simultaneously learning the dictionary and the reconstruction as part of the optimization algorithm. This has the advantage of not requiring any ground-truth reconstructions for use as training data for learning a dictionary, but the integration into the dictionary learning process imposes some practical constraints on how the convolutional representation is exploited, and it is reasonable to expect that highly under-determined problems may benefit from a pre-trained dictionary if suitable training data are available. In this paper, we propose a tomographic inversion algorithm based on the CSR model, using a dictionary learnt from an external database [24]. Instead of directly integrating the inversion into a CSC problem, which would retain some of the difficulties that have to be addressed in the CDL-based approach discussed above, we use the Plug-and-Play (PnP) framework [25, 26] to couple the tomographic inversion with a CSC model that plays the role of a Gaussian white-noise denoiser.

2 CSR for Image Denoising

One of the simplest computational imaging problem is that of recovering an image corrupted by additive white Gaussian noise. As a result, this simple inverse problem serves as a convenient test-bed for the development of new regularizers before extending them to other inverse problems. This extension is greatly simplified by the PnP method, which provides a simple method to integrate complex models expressed via denoising algorithms into the model-based inversion framework [25, 26]. Here we summarize different approaches to using CSR for solving the white-noise denoising problem.

A convolutional dictionary, , is typically learnt from a set of images by minimizing

such that , where is the high-pass component of the image, is the dictionary element, is the coefficient map corresponding to image and dictionary element , is a regularization parameter that controls the sparsity of the coefficient maps, and is a convolution operator. The dictionaries are learned from high-pass filtered training images rather than the original training images due to the difficulty in representing the low-pass components via convolutional sparse representations [27, Sec. 3]. The high-pass component is typically set as , where is a 2-D finite difference operator and controls the strength of the filter [28]. There are several algorithms to efficiently solve the CDL problem [12, 14, 15, 13, 11].

We consider three different variants of the CSC problem for white-noise denoising. The first approach (henceforth referred to as CSC-I) corresponds to the standard CSC problem, based on minimizing the function


where is the high-pass component of the noisy data , computed in the same way as for dictionary learning. The final reconstruction is obtained as where is the low-pass component of the noisy input image.

Since the standard CSC problem does not provide competitive performance in Gaussian white-noise denoising problems, we introduce a simple weighting scheme that has been found to significantly improve performance in this application, making it competitive with more well-established patch-based sparse representation methods [29]. This variant (henceforth referred to as CSC-II) can be expressed as the minimization of


where represents point-wise multiplication, and are weights that are set as


where is the matrix version of the convolutional dictionary and denotes point-wise division [29].

While the need for pre-processing of the input images is not problematic when solving a simple denoising problem, it greatly complicates direct integration of the CSC model with a more complex inverse problem since the input images and reconstructions are in different spaces. The final variant we consider avoids the need for high-pass filtering pre-processing of the input images by jointly estimating low-pass and high-pass components, using an additional regularization term that penalises the gradient of the low-pass component [27]. This problem (henceforth referred to as CSC-III) can be expressed as the minimization of


where and are algorithm parameters. The final reconstruction is obtained as . While the use of the approach of CSC-III or variants thereof is necessary when directly integrating the CSR model with tomographic inversion [23], the decoupling provided by the PnP approach makes it optional rather than essential.

3 CSR for Tomography

To leverage the idea underlying the weighted convolutional sparse coding based denoising of (2) for tomography, we utilize the Plug-and-Play priors framework [25]. The framework was originally inspired by solving a regularized inversion using the idea of variable splitting followed by use of the alternating direction method of multipliers algorithm that results in iteratively solving two sub-problems corresponding to an inversion step followed by a denoising step. Furthermore, it was empirically observed that the algorithm converges to a fixed point even if arbitrary denoisers are used in the iterative framework. Specifically, the PnP reconstruction is obtained by iterating over the steps

where corresponds to an optimization problem corresponding to the forward model, is a denoising algorithm, and and are algorithm parameters. In particular, for conventional tomography problems, is given by


where is a vector of tomographic projection measurements, is the projection matrix, and is a diagonal matrix of weights corresponding to the inverse variance of the noise. While solving (5) is expensive, in practice we partially solve it using a few iterations of an iterative algorithm.

The denoiser corresponding to CSC-II (Eq. (2)), , is given by

Notice that even though these sequence of steps do not correspond to solving an optimization problem, the PnP method allows for CSC-II (and by extension the CSR model) to be used for tomographic inversion.

4 Results

Figure 1: Training images and corresponding learnt convolutional dictionary.
Table 1: Comparison of the denoising performance of the three convolutional sparse coding based denoisers in Section 2 on measurements with different noise levels. All values represent PSNR values in units of dB.
Figure 2: Illustration of denoising performance using the three different convolutional sparse coding based methods in Section 2. Notice the weighted scheme (CSC-II) produces a high quality reconstruction and has a similar computational complexity as CSC-I.
PSNR 26 dB 20 dB 14 dB
256 25.07 | 21.41 | 24.84 23.23 | 20.68 | 23.87 22.17 | 20.07 | 22.57
128 22.18 | 20.33 | 22.99 21.07 | 19.80 | 21.96 20.06 | 19.06 | 20.68
64 18.71 | 18.09 | 20.79 18.10 | 17.69 | 19.42 17.22 | 17.05 | 17.87
Table 2: Comparison of the PSNR of tomographic reconstructions with respect to the original phantom for different levels of noise.
PSNR 26 dB 20 dB 14 dB
70 28.43 | 28.12 | 28.60 26.64 | 27.82 | 27.41 25.24 | 27.70 | 26.21
Table 3: Comparison of the PSNR of the limited-angle tomographic reconstruction with respect to the original phantom for various levels of noise.

In order to test the proposed algorithm, we use phantoms from the tomo-bank database [24]. Fig. 1 shows the nine images from the database that we used to train a multi-scale convolutional dictionary with 128 elements of sizes , , and . We use the SPORCO package [30, 28] for implementing the CDL and CSC algorithms using a value of for pre-processing the images. First, we test the different denoising strategies discussed in Section 2. Table. 1 and Fig. 2 shows the results of different denoising strategies using an image from the database that is not in the training set. Notice that the weighted scheme (CSC-II) offers superior performance across different noise levels making it a useful method for using CSR as a regularizer for inverse problems. More importantly, CSC-II has the simplicity of CSC-I and offers computational savings compared to CSC-III along with better performance.

Figure 3: Tomographic reconstruction using different algorithms for sparse-view and noisy data corresponding to the phantom in Fig. 2. The inset is a zoomed in section from the center of the reconstruction.
Figure 4: Ground-truth image and limited-angle tomographic reconstructions using different algorithms.

Next, we test the performance of the proposed algorithm on tomographic data sets. We compare the proposed algorithm to a model-based algorithm using the edge-preserving Markov-random field (MRF) regularizer [1] (with ) and a patch-based dictionary learning regularizer with a dictionary of elements of size learnt from the database. The patch model is used for tomography using the PnP framework with appropriate averaging carried out to perform the image denoising [28] and referred to in the results as patch-based sparse coding (PSC). In all cases the parameters are chosen for the highest PSNR in the reconstruction. The tomographic data was generated by projecting the phantom in Fig. 2 at , and views with three different levels of Gaussian noise corresponding to a PSNR of dB, dB and dB in the projection domain. The tomographic projection and back-projection was implemented using ASTRA tool-box [31]. For the sub-problem in the PnP framework, we use iterations of the optimized-gradient method [32] and for the sub-problem we set the maximum number of iterations to . The total number of outer-iterations is set to . Fig.3 and Table 2 shows the reconstructed results from different scenarios. Notice that at the low-noise and large-views case the MRF model outperforms both the PSC and the CSC models. However, for the sparse view and noisy data the CSC model significantly outperforms the MRF and the PSC model. Finally, we test the proposed algorithm on a limited angle noisy data set, motivated by applications such as electron tomography [1]. The data was generated by projecting the phantom shown in Fig. 4 at angles between and at the three noise levels as in the first case. Fig. 4 and Table 3 highlight that the proposed algorithm performs better than the MRF model in each case and is competitive to the PSC model for the high-noise cases.

5 Conclusion

We have presented an algorithm for using the convolutional sparse representation model as a regularizer for solving tomographic inverse problems. To overcome the potentially poor performance of conventional CSR based approaches we use a data-adaptive weighting based denoising with the plug-and-play framework for the tomographic inversion. This weighting is simple yet vital to boosting the performance of the CSR model compared to the conventional edge-preserving model and the patch-based sparse representation model.


  • [1] S. Venkatakrishnan, L. Drummy, M. Jackson, M. De Graef, J. Simmons, and C. Bouman, “A model based iterative reconstruction algorithm for high angle annular dark field - scanning transmission electron microscope (HAADF-STEM) tomography,” IEEE Trans. on Image Processing, vol. 22, no. 11, Nov. 2013.
  • [2] K. Sauer and C. Bouman, “Bayesian Estimation of Transmission Tomograms Using Segmentation Based Optimization,” IEEE Trans. on Nuclear Science, vol. 39, pp. 1144–1152, 1992.
  • [3] H. Zhang, J. Ma, Y. Liu, H. Han, L. Li, J. Wang, and Z. Liang, “Nonlocal means-based regularizations for statistical CT reconstruction,” in Medical Imaging 2014: Physics of Medical Imaging, 2014, vol. 9033, p. 903337.
  • [4] Q. Xu, H. Yu, X. Mou, L. Zhang, J. Hsieh, and G. Wang, “Low-dose X-ray CT reconstruction via dictionary learning,” IEEE Trans. on Medical Imaging, vol. 31, no. 9, pp. 1682–1697, 2012.
  • [5] R. Zhang, D. H. Ye, D. Pal, J.-B. Thibault, K. D. Sauer, and C. A. Bouman, “A Gaussian mixture MRF for model-based iterative reconstruction with applications to low-dose X-ray CT,” IEEE Trans. on Computational Imaging, vol. 2, no. 3, pp. 359–374, 2016.
  • [6] X. Zheng, Z. Lu, S. Ravishankar, Y. Long, and J. A. Fessler, “Low dose CT image reconstruction with learned sparsifying transform,” in Proc. of the IEEE Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), July 2016.
  • [7] J. Luo, H. Eri, A. Can, S. Ramani, L. Fu, and B. De Man, “2.5D dictionary learning based computed tomography reconstruction,” in Anomaly Detection and Imaging with X-Rays (ADIX), 2016, vol. 9847, p. 98470L.
  • [8] M. S. Lewicki and T. J. Sejnowski, “Coding time-varying signals using sparse, shift-invariant representations,” in Advances in neural information processing systems, 1999, pp. 730–736.
  • [9] M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 2528–2535.
  • [10] B. Wohlberg, “Efficient algorithms for convolutional sparse representations,” IEEE Trans. on Image Processing, vol. 25, no. 1, pp. 301–315, Jan. 2016.
  • [11] C. Garcia-Cardona and B. Wohlberg, “Convolutional dictionary learning: A comparative review and new algorithms,” IEEE Trans. on Computational Imaging, vol. 4, no. 3, pp. 366–381, 2018.
  • [12] V. Papyan, Y. Romano, M. Elad, and J. Sulam, “Convolutional dictionary learning via local processing.,” in Proc. of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5306–5314.
  • [13] I. Y. Chun and J. A. Fessler, “Convolutional dictionary learning: Acceleration and convergence,” IEEE Trans. on Image Processing, vol. 27, no. 4, pp. 1697–1712, 2018.
  • [14] K. Degraux, U. S. Kamilov, P. T. Boufounos, and D. Liu, “Online convolutional dictionary learning for multimodal imaging,” in Proc. of the IEEE International Conference on Image Processing (ICIP), 2017, pp. 1617–1621.
  • [15] J. Liu, C. Garcia-Cardona, B. Wohlberg, and W. Yin, “First and second order methods for online convolutional dictionary learning,” SIAM Journal on Imaging Sciences, vol. 11, no. 2, pp. 1589–1628, 2018.
  • [16] B. Wohlberg, “Efficient convolutional sparse coding,” in Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Florence, Italy, May 2014, pp. 7173–7177.
  • [17] F. Heide, W. Heidrich, and G. Wetzstein, “Fast and flexible convolutional sparse coding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 5135–5143.
  • [18] S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng, and L. Zhang, “Convolutional sparse coding for image super-resolution,” in Proc. of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1823–1831.
  • [19] Y. Liu, X. Chen, R. K. Ward, and Z. J. Wang, “Image fusion with convolutional sparse representation,” IEEE Signal Processing Letters, vol. 23, no. 12, pp. 1882–1886, 2016.
  • [20] A. Serrano, F. Heide, D. Gutierrez, G. Wetzstein, and B. Masia, “Convolutional sparse coding for high dynamic range imaging,” Computer Graphics Forum, vol. 35, no. 2, pp. 153–163, May 2016.
  • [21] H. Zhang and V. M. Patel, “Convolutional sparse and low-rank coding-based rain streak removal,” in Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), March 2017.
  • [22] T. M. Quan and W.-K. Jeong, “Compressed sensing reconstruction of dynamic contrast enhanced MRI using GPU-accelerated convolutional sparse coding,” in Proc. of the IEEE International Symposium on Biomedical Imaging (ISBI), 2016, pp. 518–521.
  • [23] E. Skau and C. Garcia-Cardona, “Tomographic reconstruction via 3D convolutional dictionary learning,” in Proc. of the IEEE Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), June 2018.
  • [24] F. De Carlo, D. Gürsoy, D. J. Ching, K. J. Batenburg, W. Ludwig, L. Mancini, F. Marone, R. Mokso, D. M. Pelt, J. Sijbers, et al., “Tomobank: a tomographic data repository for computational X-ray science,” Measurement Science and Technology, vol. 29, no. 3, pp. 034004, 2018.
  • [25] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in Proc. of IEEE Global Conference on Signal and Information Processing (GlobalSIP), Austin, TX, USA, Dec. 2013.
  • [26] S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, “Plug-and-play priors for bright field electron tomography and sparse interpolation,” IEEE Trans. on Computational Imaging, vol. 2, no. 4, pp. 408–423, Dec. 2016.
  • [27] B. Wohlberg, “Convolutional sparse representations as an image model for impulse noise restoration,” in Proc. of the IEEE Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Bordeaux, France, July 2016.
  • [28] B. Wohlberg, “SPORCO: A Python package for standard and convolutional sparse representations,” in Proc. of the 15th Python in Science Conference, Austin, TX, USA, July 2017, pp. 1–8.
  • [29] B. Wohlberg, “Convolutional sparse coding with overlapping group norms,” Tech. Rep. 1708.09038, arXiv, Aug. 2017.
  • [30] B. Wohlberg, “SParse Optimization Research COde (SPORCO),” Software library available from, 2016.
  • [31] F. Bleichrodt, T. van Leeuwen, W. J. Palenstijn, W. van Aarle, J. Sijbers, and K. J. Batenburg, “Easy implementation of advanced tomography algorithms using the ASTRA toolbox with Spot operators,” Numerical Algorithms, vol. 71, no. 3, pp. 673–697, March 2016.
  • [32] D. Kim and J. A. Fessler, “An optimized first-order method for image restoration,” in Proc. of the IEEE International Conference on Image Processing (ICIP), Sept. 2015, pp. 3675–3679.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description