Generating Training Data for Denoising
Real RGB Images via Camera Pipeline Simulation

Most image reconstruction techniques such as denoising operate on RGB images, either JPEGs directly from a camera or RAW files that have been demosaicked later. In this paper, we show that the simple additive Gaussian noise (AWGN) usually used in the literature [33, 22, 32] does not accurately model the artifacts observed in real image. This is especially true when working from JPEG images, which undergo a long pipeline that includes operations such as demosaicking, denoising, and compression that dramatically transform the noise (see Fig. 2). The mismatch of noise profiles can have a strong adverse effect on performance, especially for learning-based approaches.

Several works have shown that image reconstruction tasks can benefit from better noise modeling [4, 9, 5, 13]. However, most noise and degradation models found in the literature remain simplistic. For example, most works do not consider demosaicking artifacts, and the ones that do typically use bilinear demosaicking [5, 13, 25], which is rarely used in real consumer cameras [15, 14].

In this paper, we propose a camera simulation pipeline that can be used to realistically simulate camera processing of images. We implement over 40 individual modules that can be custom-built into a camera pipeline. While not exhaustive, they cover a good range of typical camera modules such as tonemapping, demosaicking, and denoising. From these modules, we build a pipeline capable of processing RAW images, with some manual tuning, into visually similar RGB images that some cellphone cameras produce.

We believe this pipeline can be used to generate data for many low-level vision tasks. To demonstrate, we use our pipeline to study the importance of noise modeling in supervised denoising of JPEG images. We generate different versions of a dataset, where the pipeline processing parameters and, therefore, the noise characteristics vary. We train state-of-the-art denoising convolutional neural networks (CNN) [26] on these datasets and measure their performance on real processed images. We show that networks trained with our realistic pipeline outperformed ones that are trained on AWGN by roughly 3 dB on real test images (see Fig. 1 and Fig. 7).

To understand how our pipeline contributes to this improvement, we train additional networks with different combinations of simulation components. We find that performance drops markedly if denoising and/or demosaicking components are removed. Furthermore, the choice of demosaicking algorithm is also important. Using bilinear demosaicking in camera simulation pipelines [5, 13, 25] leads to less effective denoising compared to using edge-aware methods such as the Adaptive Homogeneity-Directed (AHD) algorithm [17].

While we are not the first to propose a camera simulation [19, 11], our main contribution is to integrate it into the learning pipeline and use it to show the importance of realistic simulation for learning-based image restoration tasks. We summarize our contributions as follow:

1. We propose a camera simulator that is expressive enough to simulate processing of real cameras. We believe that this simulator will be useful for many learning-based image restoration tasks.

2. Using this simulator, we study the importance of realistic noise modeling for denoising real images. We show that:

   1. A realistic noise model is beneficial for denoising real JPEG images and leads to superior performance compared to AWGN.

   2. Denoising filters and demosaicking are the most important components for simulating realistic noise.

   3. Realistic demosaicking algorithm is important and leads to improvement over bilinear demosaicking commonly used in camera simulation pipelines [5, 13, 25].

All code and evaluation datasets will be released publicly.

Classical denoising techniques often create probabilistic models of the noise and signal and use this model to derive a denoising algorithm. Wavelet coring is based on the observation that noise is usually smaller than the image signal, resulting in smaller wavelet coefficients that can be suppressed [29, 27, 10]. The current state-of-the-art classical denoising method remains BM3D [8]. The algorithm performs non-local matching within the image and average these matched blocks together. These methods typically assume an AWGN model in order to simplify their modeling effort.

With the growing popularity of CNNs [21], learning-based denoising is becoming prevalent. DnCNN [32] uses CNNs to predict a residual map that corrects noisy images. N3Net [26] formulates a differentiable version of nearest neighbor search to further improve DnCNN. FFDNet [33] attempts to address spatially varying noise by appending noise level maps to the input of DnCNN. Despite many improvements, these works perform very similarly, with often less than 0.5 dB difference. Moreover, they assume spatially uncorrelated noise, which is not true for real noisy JPEG images.

Many works in image processing are recognizing the important of noise modeling. [12, 15] jointly model noise with their demosaicking task and found it to improve their performance. [4] uses an adhoc noise model that simulates spatial correlation of noise. They found this to significantly boost the quality of their deblurring results.

Recent denoising work proposes simulating camera pipelines. [5] unprocesses JPEG images to get RAW representation and focus on RAW-to-RAW denoising. Very related to our work is [13] who also uses simulated camera pipeline to supplement real training data. However, these works tend to assume a limited camera pipeline and do not evaluate on real processed images. Our work follows in the same spirit, though we aim to accurately model realistic camera pipelines, and evaluate our results with real images.

Our camera pipeline is designed to mirror typical camera processing stages. We build our pipeline from individual modules that are easily extensible. Additionally, we also include an artifact generation stage that simulates the degradation of the image signal, by introducing artifacts such as noise and motion blur.

Fig. 3 shows the 4 main stages of our pipeline: artifact simulation, demosaicking, denoising, and postprocessing. In sum, we have over 70 parameters that control the behavior of our pipeline. We describe each stage as follows.

Artifact Generation. The first stage of our pipeline is the physical artifact simulators. It aims to simulate the physical degradation process that happens before the sensor. It includes motion blur, chromatic aberration, multiplicative exposure adjustment, and noise.

Noise at the sensor is largely uncorrelated and zero-mean. So we only simulate spatially uncorrelated noise here. Because photon noise is Poisson in nature and the sensor read noise is Gaussian, we provide both additive and multiplicative noise to simulate the two effects. We optionally mosaick input images before adding noise if the user wishes to simulate the Bayer pattern, which will then be demosaicked and processed in the subsequent stages.

Demosaicking. If the input is mosaicked, we demosaick the input at this stage. To our knowledge, most cameras use more advanced algorithms, such as the Adaptive Homogeneity-Directed (AHD) algorithm [17]. We provide a Python adaptation of the reference AHD algorithm and an algorithm developed by Kodak Inc. [16] implemented in high performance language Halide [23]. We provide bilinear demosaicking as well because it has been widely used in the recent camera simulation literature [5, 25, 13]. Hot/Dead pixel correction and white balance also occur here.

Cellphone Denoising. Demosaicking noisy images tends to result in long-grained artifacts (as Fig. 2 shows). Our third stage applies denoising to the image. We include three denoising algorithms–bilateral filters [30], median filters [18], and wavelet coring [10]–with the option to turn each algortihm on/off as well as reorder them. Performing tonemapping prior to denoising can be beneficial for denoising different intensity ranges because it can non-linearly compress a particular range of the intensity leading to a more smoothing effect. We include a pre-tonemapping operator, which can be a gamma or an s-shape tone curve.

Tonemapping and Post-processing. The last stage performs postprocessing that aims to generally improve the aesthetics of the image. We include saturation adjustment, tonemapping for additional tone/contrast enhancement, unsharp mask for detail enhancement, and JPEG compression for JPEG compression artifacts simulation.

We build our pipeline largely on top of the PyTorch package [24]. This allows us to readily integrate it into learning frameworks. Because some of the software used does not support differentiation [31], we do not utilize the differentiability of our pipeline. While we believe that our pipeline is realistic and rich in features, it is by no means a comprehensive set of operations implemented by camera manufacturers. In particular, we do not consider automatic adjustments such as auto-exposure and auto white-balance. These modules will become crucial in automatic processing of cellphone images. Nonetheless, we demonstrate in section 3.1 that our pipeline can emulate cellphone processing, given an appropriate set of parameters.

We demonstrate that our pipeline can be used for generating training data for real image denoising. Denoising is a well-studied topic, yet, few works have attempted to model realistic noise correlation. We show that the lack of realistic noise can be detrimental to denoising performance.

We synthetically generate our datasets using the camera simulation pipeline described in Section 3. Using different sets of parameters, we seek to answer two important questions: does having a realistic noise model matter, and if so, how realistic does the noise model have to be?

Many works on denoising are shifting towards denoising RAW images, where noise is easier to model [12, 5, 7]. We focus on denoising JPEG images for two reasons. First, most image reconstruction algorithms deal primarily with JPEG images. But for these methods, using an additive white Gaussian noise model with JPEG images can lead to inferior results [4] . Second, many photographs taken are in JPEG format because it is often easier to work with and uses less storage. Therefore, any algorithm that aims to be widely adopted must be able to deal with the degradation present in the JPEG images.

We primarily focus on learning-based approaches, for which synthetic data generation is useful for training. Methods that do not require training data may still find it useful to generate realistic test data as an alternative to collecting their own dataset.

In this Section, we report the findings of our denoising experiments.

We have proposed a realistic camera pipeline simulation that is expressive enough to process RAW inputs into JPEG images that is visually similar to the ones cameras produce. We use this simulation to generate realistic datasets for training denoising CNNs and show that it improves the performance of such networks on real JPEG images by over 3dB. Demosaicking and denoising seem to be the most important components of our pipeline that enable such improvement. Removing either of them leads to a significant drop in the quality of the denoised output. Using correct algorithms for these components is also important. The bilinear demosaicking algorithm commonly used in previous camera simulation work [5, 13, 25] leads to a significant performance drop, while edge-aware algorithms such as AHD [17] do not.

While we have shown our pipeline is useful and realistic, it still requires significant manual tuning in order to match the appearance of the processed JPEG. The ability to automatically match the appearance is an interesting future direction. This will help ensure realism, so that the generated data can be used for any arbitrary camera models.

The authors would like to thank the Toyota Research Institute for their generous support of the projects. We thank Tzu-Mao Li for his helpful comments, and Luke Anderson for his help revising this draft.