Concise Radiometric Calibration Using The Power of Ranking

Concise Radiometric Calibration Using The Power of Ranking

Abstract

Compared with raw images, the more common JPEG images are less useful for machine vision algorithms and professional photographers because JPEG-sRGB does not preserve a linear relation between pixel values and the light measured from the scene. A camera is said to be radiometrically calibrated if there is a computational model which can predict how the raw linear sensor image is mapped to the corresponding rendered image (e.g\bmvaOneDotJPEGs) and vice versa. This paper begins with the observation that the rank order of pixel values are mostly preserved post colour correction. We show that this observation is the key to solving for the whole camera pipeline (colour correction, tone and gamut mapping). Our rank-based calibration method is simpler than the prior art and so is parametrised by fewer variables which, concomitantly, can be solved for using less calibration data. Another advantage is that we can derive the camera pipeline from a single pair of raw-JPEG images. Experiments demonstrate that our method delivers state-of-the-art results (especially for the most interesting case of JPEG to raw).

\addauthor

Han Gongh.gong@uea.ac.uk1 \addauthorGraham D. Finlaysong.finlayson@uea.ac.uk1 \addauthorMaryam M. Darrodi1 \addinstitution School of Computing Sciences
University of East Anglia
Norwich, UK
denotes co-first author Concise Radiometric Calibration by Ranking

1 Introduction

Many computer vision algorithms (e.g\bmvaOneDotphotometric stereo [Petrov(1993)], photometric invariants [Gevers et al.(2004)Gevers, Finlayson, and Schettini], shadow removal [Gong and Cosker(2016)], and colour constancy [Barnard et al.(2002)Barnard, Martin, Coath, and Funt]) assume that the captured RGBs in images are linearly related to the actual scene radiance. However, the imaging pipeline in a digital camera is necessarily non-linear in order to produce perceptually-pleasing photos rather than its physically-meaningful counterparts. In this paper, we present a new rank-based radiometric calibration method which solves for the bi-directional mappings between the camera’s RAW responses and the rendered RGBs produced by digital camera.

There is prior art in this field which models the pipeline with a large number of parameters (up to several thousand [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko]) which both means a large corpus of data is required to uncover the pipeline and that there is at least tacitly the premise that the underlying pipeline is quite complex. The key insight in our approach is that post-colour correction (a matrix correction) the linear corrected raw RGBs are to the greatest extent in the same rank order as the final rendered RGBs. Building on this insight, we develop a simple rank-based radiometric calibration model that “solves for” the camera pipeline with many fewer parameters and concomitantly needs much less training data.

In Fig. 1, we illustrate a conventional image reproduction pipeline that holds for many cameras [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown]. An exemplar raw image, Fig. 1a, is mapped by a colour correction matrix to give the image shown in Fig. 1b. The colour correction matrix implements several processing steps (e.g\bmvaOneDotillumination correction [Ramanath et al.(2005)Ramanath, Snyder, Yoo, and Drew, Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler], display RGB mapping [Anderson et al.(1995)Anderson, Motta, Chandrasekar, and Stokes], and colour preference adjustments [Ramanath et al.(2005)Ramanath, Snyder, Yoo, and Drew]). It is well-known that a display device cannot display all captured image colours that some RGBs will fall outside the RGB cube after mapping (e.g\bmvaOneDotthe pixels marked in purple in Fig. 1b). We therefore need gamut mapping, e.g\bmvaOneDot[Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown, Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko, Giesen et al.(2007)Giesen, Schuberth, Simon, Zolliker, and Zweifel], to bring the colours back inside the cube as shown in Fig. 1c. Finally, the gamut mapped image is tone mapped to arrive at the final rendered output [Ramanath et al.(2005)Ramanath, Snyder, Yoo, and Drew, Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler, Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] shown in Fig. 1d. Tone mapping accounts for the display non-linearity [Anderson et al.(1995)Anderson, Motta, Chandrasekar, and Stokes], dynamic range compression and some aspects of preference [Reinhard et al.(2002)Reinhard, Stark, Shirley, and Ferwerda].

Figure 1: a) a RAW input image is colour corrected to give image b). Non-displayable colours are highlighted in purple pseudo colour. Gamut mapping, in step c), brings colours within gamut. Finally, in d), a tone mapping step results in the final rendered image.

The colour processing pipeline – for cameras, in general, can be written as Eqn. 1.

(1)

Here and throughout this paper, denotes a camera RAW and refers to its rendered RGB counterpart. Respectively, the correction matrix, gamut mapping and tone mapping are denoted by the matrix and the functions and . The function can implement a single or three per-channel tone curves. Because gamut mapping only implements a small change in comparison with colour and tone mapping steps, the order of gamut mapping and tone mapping may be switched (Eqn. 1b & c), a property that we exploit in this paper. Equally, we can also roll all three processing steps into one and directly solve for a 3D LUT (Look-Up-Table) that maps RAW to rendered counterparts. This LUT function is denoted  [Lin et al.(2012)Lin, Lu, Kim, and Brown] in Eqn. 1c. Readers may refer to the top row of Fig. 1 to link each mathematical function to our example processed image.

In radiometric calibration, given a set of and , one seeks to solve for the parametrised pipeline parts (e.g\bmvaOneDot, , and ). A disadvantage of the current best performing methods is that a great deal of data may be required to fit their assumed models. In Eqns. 1a and 1b, the gamut mapping step could be modelled by 1000s of Radial Basis functions [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown, Lin et al.(2012)Lin, Lu, Kim, and Brown, Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] and in Eqn. 1c, the deployed LUT could also have several thousand control points.

Our proposed method begins with the simple observation [Finlayson et al.(2016)Finlayson, Darrodi, and Mackiewicz] that, assuming the gamut mapping step makes small changes to image colours, we expect the rank ordering of the rendered to be the same as multiplied by the correction matrix (because a tone curves are always monotonically increasing). Suppose that two rendered (JPEG) responses – in the 1 red colour channel – are denoted and ; and that . The rank order of two corresponding raw red channel measurements post colour correction is written as (where denotes the first row of and and are a pair of raw RGBs). Rewriting: this implies that which mathematically defines a half-space constraint. If we visualise the row vector as a point in 3-space then this inequality – which we call a ranking constraint – forces the point to be located in one half of 3-space but not the other. Because we have multiple pixels, each pair of pixels (2 raw and 2 JPEG RGBs) generates a half space constraint and intersecting all these constraints delimits the region in which must lie. Our experiments demonstrates that a small numbers of patches suffices to estimate accurately. Once we have we then find the best rank preserving tone curves . At this stage, only using and we have a tolerable approximation of the pipeline. Indeed, we argue that our construction of and also incorporates, to a first order, gamut mapping. Now we adopt (Eqn 1c) and find a 125-parameter LUT to “mop up” any remaining errors due to gamut mapping (higher order terms).

2 Related work

Using the pipeline form Eqn 1b, Chakrabarti et al\bmvaOneDot [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] first solve for and (in a least-squares sense) in iteration and then solve directly for . In their approach, is constrained to be a 7 order monotonic polynomial. They model with the radial basis function (RBF) method of [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] where several thousands of RBFs are potentially used. A restriction of the above calibration is presented in [Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler] where the gamut mapping is ignored. This less general model works tolerably well on many real pairs of raw and rendered images and this is a point we will return to later in this paper. In either version ([Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] or [Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler]), the coupled nature of the minimization means that a global minimum is not guaranteed to be found. Thus, a random start search is incorporated – multiple minimisations are carried out – in order to find their best parameters. Kim et al\bmvaOneDot[Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] solve for the pipeline in the form of Eqn. 1a and makes additional assumptions to decouple the optimization. They assume that images of the same scene are captured with respect to two or more exposures and their is a multi-thousand set of RBFs. Regarding solving for , Debevec et al\bmvaOneDot [Debevec and Malik(1997)] showed how relating corresponding pixels under known exposure differences suffices to solve for (assuming there is no gamut mapping step). Importantly, in [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown], it was argued that for the set of desaturated pixels (i.e\bmvaOneDotRAWs far from the RGB cube boundary) the gamut mapping step has little or no effect and can be ignored. Relative to this assumption, can be solved using the Debevec method. Given then the colour correction matrix can be found (again using desaturated pixels). Though, for typical capture conditions, e.g. for most mobile phones, multiple exposures are not available and so the requirement that multiple exposures are needed is a weakness in this method. Finally, in [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] a gamut mapping RBF network is “trained”. Of course, if a large number of radial basis functions are used to model gamut mapping (as proposed in [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] or [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko]) then solving for requires a large corpus of data. Further the application of gamut mapping is expensive and its parametrisation is large.

In [Lin et al.(2012)Lin, Lu, Kim, and Brown] it was shown that is possible to ignore the underlying structure of the colour processing pipeline and directly solve for the best 3D surjective function – implemented as a LUT that maps the RAWs to rendered RGBs (Eqn. 1c). Finally, in [Lin et al.(2004)Lin, Gu, Yamazaki, and Shum], a method is presented for solving for by examining the edge distribution in an image. This method has the advantage that the method works for a single image (no need for multiple exposures) but the disadvantage that the method is sensitive to processing steps such as image sharpening which is used extensively in mobile phone image processing.

3 The rank-based method

As the reader shall see, to make the rank-based method work we need to assume that the gamut mapping step only makes small adjustments to colour. In fact our assumption is more nuanced. We assume that – to a first order – gamut mapping can mostly be implemented as an affine transform and that this affine transform can be folded into the colour correction matrix and the monotonically increasing tone mapping functions .

3.1 Gamut Mapping as an Affine Transform

In Eqn. 1b, gamut mapping is applied when, after colour correction, colours are mapped outside the colour cube and become non-displayable. Let us use a Taylor expansion to model around a point inside the gamut:

(2)

where is the Jacobian (matrix of derivatives of ). Not only does Eqn. 2 show that, to a first approximation, gamut mapping is an affine transform it is also one of the gamut mapping algorithms proposed in [Giesen et al.(2007)Giesen, Schuberth, Simon, Zolliker, and Zweifel]. In particular, [Giesen et al.(2007)Giesen, Schuberth, Simon, Zolliker, and Zweifel] solves, with good results, for the best affine transform that maps image colours inside the gamut and which are, simultaneously, close to the non-gamut mapped colours:

(3)

In Eqn. 3, and are respectively a matrix and offset vector defining the affine gamut mapping algorithm. The 3-vectors of 0s and 1s are denoted and Eqn. 3 is solved directly by Quadratic Programming [Gill et al.(1981)Gill, Murray, and Wright]. The gamut mapping shown in Fig. 1c is the result of solving Eqn. 3.

Here, we make two important remarks about affine gamut mapping: 1) Gamut mapping and colour correction combined can be represented by the single affine transform: matrix and offset ; 2) It follows that the rank-based method presented in the next section will actually solve for . The offset term can be incorporated directly in (since an offset does not change ranking).

3.2 Rank-based estimation for colour correction

Let us denote the row of as , let us assume that given two colour corrected RAWs, and that the rank order is the same as for the corresponding rendered RGBs:

(4)

Defining the difference vector :

(5)

where it is understood the superscript denotes the difference vector from the of pairs of image pixel values. Suppose that we have a vector where Eqn. 5 holds, then the inequality cannot be true for . That is Eqn. 5 defines a half plane constraint [Finlayson et al.(2016)Finlayson, Darrodi, and Mackiewicz, Berg et al.(2008)Berg, Cheong, Kreveld, and Overmars]. The vector is perpendicular to the half-plane: any less than 90 degrees to is a possible solution. Given multiple difference vectors then we have multiple half-plane constraints which taken together delimit a region in 3-space where must lie. Denoting the half-plane as , must satisfy:

(6)

Let us visualise the computation of using ranking. Without loss of generality let us assume that . We rewrite Eqn. 5 as

(7)

If is a solution to Eqn. 6, then for Eqn. 7 is also true since and . Solutions for lie on one side of the line, i.e\bmvaOneDotthe 3D half-space constraints maps directly to a 2D half-plane constraint. Or, if we consider the whole set of collations, the cone in 3D, defined by Eqn. 6, maps to a 2D convex region [Finlayson(1996)]. Denoting half-planes as we, equivalently, solve for

(8)

The intersection problem of Eqn. 8 is easily visualised. In Fig. 2a we show the intersection of 4 half plane constraints and indicate the solution set where must lie.

Figure 2: a) The region where 4 half-plane constraints intersect delimit the region where must lie where the black point is a feasible solution. b) On an unit sphere, each vector represented by the origin and a blue surface point is a probe for a possible solution (e.g\bmvaOneDotthe black arrow). All 3D points and constraints are projected to a 2D plane .

We solve for one sensor channel at a time. Empirically, we have to be careful not to generate too many half planes. In our experiment, we generate half-planes from all pairs of up to 50 randomly selected unique RAWs, generating half-planes. Due to noise or small deviations in real camera data, it is likely that no common intersection can be found that satisfy every half-planes constraint. To solve this problem, we generate 100,000 unit length vectors that are uniformly distributed on the surface of the unit sphere [Rakhmanov et al.(1994)Rakhmanov, Saff, and Zhou], which is visualised in Fig. 2b. With respect to this sampling, the furthest distance between any point and its nearest neighbour is less than 1.15 degrees. So, the orientation of the rows of are found to this accuracy. For each point on the sphere (i.e\bmvaOneDota possible row of ), we count how many half-space constraints are satisfied. The point on the unit sphere that satisfies most half-plane constraints – or the median of multiple points if there is a tie – defines . To make our approach robust, we find randomly select 50 colours 25 times and for each trial find the best . Overall, we find the that places all the corresponding raw and rendered image RGBs in the most similar rank order. That is, if we plot the mapped raw red responses, for example, against the rendered red JPEG corresponding values then the graph should be a monotonically increasing function. How well a monotonically increasing function fits our data can be used to judge the efficacy of each .

Ranking can only estimate up to an unknown scaling of its rows. Suppose for a rendered achromatic RGB and the corresponding raw , we apply: where places a vector along the diagonal of a diagonal matrix. After this step, maps achromatic colours correctly. Because (where in the example, ) we might also solve for in a least-squares sense by including all colours indexed close to the achromatic axis: (our experiment does not include this additional step).

3.3 Rank-preserving optimization of tone curves

We now solve for the optimal per-channel tone curves which map colour corrected RAWs to corresponding rendered RGBs. Let us denote the colour corrected RAW and rendered RGB pixel pairs for the channel as . Then, the -channel rank-preserving tone curve is optimised as a order monotonic and smooth polynomial function as follows:

(9)

where the first term is for data fitness, the second term is for curve smoothness and is a small weight (e.g\bmvaOneDot). This polynomial fitting is solved by Quadratic Programming [Gill et al.(1981)Gill, Murray, and Wright]. In this paper, we further denote the combination of all 3-channel mappings as .

3.4 Gamut correction step

As argued previously, we propose that has the expressive power to implement colour correction, tone correction and gamut mapping (to the first order in a Taylor expansion). However, we wish to add a further gamut mapping step for the higher order terms. But, since our hypothesis is that much of the gamut mapping will have been accounted for we are going to adopt a simple small parameter solution. Further, this additional correction is going to be carried out at the end of the process, we adopt Eqn. 1b. Specifically, we find a LUT by using lattice regression [Garcia et al.(2012)Garcia, Arora, and Gupta] that minimises .

3.5 Rank-based recovery of raw

Suppose we wish to map rendered RGBs to RAWs. Using the method presented in Section 3.2, has already been solved in the RAW-to-JPEG forward estimation phrase. Now, in a least-squares optimal way, we use the same polynomial fitting method (Eqn. 9) to find by optimising . Finally, we solve for the backward by optimising where the LUT is fitted by a lattice regression [Garcia et al.(2012)Garcia, Arora, and Gupta].

3.6 Parameter counting

Assuming we solve for 3 independent tone curves then our method requires 9 (for ) + 24 (for ) + 125 (for ) = 158 parameters which is significantly less (even an order of magnitude less) than [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko, Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown, Lin et al.(2012)Lin, Lu, Kim, and Brown].

4 Evaluation

Our evaluation is based on the most challenging dataset that we have encountered:  [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] which contains the RAW/JPEG intensity pairs of 140 colour checker patches viewed under multiple viewing conditions. Specifically, the colour chart is captured by 8 cameras (3 of which are JPEG-only) and under 16 illuminants across many different exposures.

Below, we carried out the same experiment described in [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko]. We are interested in validating whether our method, with much reduced number of parameters can produce, similar or even better results compared with [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko]. We evaluate both RAW-to-JPEG and JPEG-to-RAW. The dataset [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] captures a sort of “worst-case” viewing conditios. Normally, when we capture a picture there is a single prevailing illuminant colour. In the dataset of Chakrabarti et al\bmvaOneDot, all camera processing parameters are turned off and then the same reflectances are viewed under multiple coloured lights. As Forsyth observed [Forsyth(1990)], the reddest red camera response cannot be observed under a blue light. And, then he exploited this observation to solve for the colour of the light. Yet, in this dataset the reddest red, the greenest green and the bluest blue can all appear simultaneously in the same image. Practically, we believe the need to divine a pipeline for the all lights all surfaces case means the prior art pipelines are probably more complex than they need to be.

As described in [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko], for each camera, we estimate the parameters of a calibration model using different subsets of available RAW-JPEG pairs. For each subset and a selected camera, the root mean-squared error (RMSE) between the prediction and ground truth is validated by using all available RAW-JPEG pairs. Table 1a and Table 1c shows the RAW-to-JPEG mapping error (where pixel intensities are coded as integers in the interval . In the table, Prob denotes the Chakrabarti method (with several thousands parameters) and RB the rank-based method with 158 parameters. We found that our forward errors are close to the results of [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko], especially for the condition of less than 3 illuminants which are more likely to occur in the real world. Evidently, for the many illuminant case the prior art has a small advantage. Remembering that JPEGs are coded as integers in [0,255] the RMSE is typically 1 or less (for RB compared to Prob). Practically, when the “fits” are viewed visually (by looking at images) it has hard to see the difference between the two methods.

For computer vision, we are more interested in the performance of JPEG-to-RAW mapping which is shown in Table 1b and Table 1c. In [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko], a probabilistic framework for mapping rendered RGB to raw was presented. Here we take their mean estimates as the most likely RAW predictions. We found that our method generally reduces the errors of [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] by . Our supplementary material also includes the additional experiment results compared with “[Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler] + our LUT” for interested readers.

The reader might be interested why our simple method seems to work so well going from rendered to raw (better than [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko]) but not quite as well as the prior art in the forward direction (albeit visually almost indistinguishable). Our hypothesis here is that the LUT in the forward direction is applied post the tone curve. This curve (at least for dark values) has a very high slope and, consequently, the coarsely quantised LUT cannot capture gamut mapping well. Yet, in the reverse direction (JPEG to RAW) the LUT is applied in linear raw where a course uniform quantisation is more justified.

a) RAW-to-JPEG Uniform 8K 10 Exp. 1 Illu. 10 Exp. 2 Illu. 4 Exp. 4 Illu.
Camera Prob RB Prob RB Prob RB Prob RB
Canon_EOS_40D 1.84 2.56 9.79 10.10 7.53 4.13 4.06 5.87
Canon_G9 2.17 3.70 6.51 6.20 3.41 5.48 3.09 4.79
Canon_PowerShot_S90 2.44 3.24 4.88 4.52 3.58 4.34 3.40 4.04
Nikon_D7000 1.72 4.03 8.05 10.03 3.32 5.39 26.06 6.48
Panasonic_DMC-LX3 1.65 3.65 7.33 8.70 5.25 4.56 3.05 7.98
8 Exp. 4 Illu. 4 Exp. 6 Illu. 8 Exp. 6 Illu.
Camera Prob RB Prob RB Prob RB
Canon_EOS_40D 2.91 4.13 3.60 4.11 2.25 3.61
Canon_G9 2.79 5.48 3.12 4.74 2.77 4.67
Canon_PowerShot_S90 2.95 4.34 3.27 3.70 2.75 3.93
Nikon_D7000 2.41 5.39 2.77 5.04 1.92 4.95
Panasonic_DMC-LX3 2.77 4.56 2.94 4.26 2.33 4.14
b) JPEG-to-RAW Uniform 8K 10 Exp. 1 Illu. 10 Exp. 2 Illu. 4 Exp. 4 Illu.
Camera Prob RB Prob RB Prob RB Prob RB
Canon_EOS_40D 0.079 0.060 0.085 0.072 0.080 0.064 0.075 0.072
Canon_PowerShot_G9 0.126 0.075 0.143 0.104 0.120 0.079 0.120 0.082
Canon_PowerShot_S90 0.065 0.052 0.073 0.058 0.069 0.074 0.066 0.057
Nikon_D7000 0.143 0.090 0.543 0.123 0.140 0.098 0.229 0.108
Panasonic_DMC-LX3 0.082 0.058 0.090 0.072 0.082 0.063 0.073 0.071
8 Exp. 4 Illu. 4 Exp. 6 Illu. 8 Exp. 6 Illu.
Camera Prob RB Prob RB Prob RB
Canon_EOS_40D 0.071 0.064 0.077 0.065 0.069 0.063
Canon_PowerShot_G9 0.121 0.079 0.126 0.076 0.126 0.080
Canon_PowerShot_S90 0.069 0.074 0.063 0.059 0.066 0.058
Nikon_D7000 0.144 0.098 0.147 0.094 0.143 0.101
Panasonic_DMC-LX3 0.077 0.063 0.074 0.060 0.077 0.064
c) Uniform 8K (JPEG-Only Camera Test) RAW-to-JPEG JPEG-to-RAW
Camera Raw Proxy Prob RB Prob RB
FUJIFILM_J10 Panasonic_DMC-LX3 10.43 11.51 0.279 0.077
Galaxy_S_III Nikon_D7000 11.34 13.13 0.114 0.074
Panasonic_DMC_LZ8 Canon_PowerShot_G9 8.85 12.23 0.146 0.085
Table 1: RMSE between ground truth and prediction for bidirectional RAW and JPEG conversions: Prob denotes [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] and RB is our rank-based method. “Exp.” and “Illu.” are respectively short for “Exposure” and “Illuminant”. “Raw Proxy” is the camera used to capture raw for the camera which does not support raw image capturing.

5 Calibration with small numbers of parameters

We wished to visually validate our claim that we can calibrate with few parameters. We took 4 RAW+JPEG pairs (for different cameras) from [Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler]. We then uniformly selected 140 corresponding pixels from the RAW and JPEG. We solved for all the 158 parameters in our rank-based method. We then applied our model to the rest of the image. The result of this experiment for 4 images (JPEG-to-RAW) is shown in Fig. 3.

Figure 3: Visualisation of one-shot radiometric calibration through a simulated 140-patch colour checker, shown at the top-right corner of each Rendered JPEG image. The error maps in the 4 and 5 columns respectively visualise the per pixel RMSE for our rank-based method with & without the gamut mapping LUT. The RMSE of each whole image is shown at the top-right corner of each error map. All raw images are shown with a 0.5 gamma.

6 Conclusion

In this paper we have shown how the rank order of image responses is a powerful tool for solving for the individual steps in a camera processing pipeline (colour correction, gamut and tone mapping). A simple ranking argument, relating colour corrected RAWs to corresponding rendered RGBs suffices to solve for the colour correction matrix. Then, the rank-preserving tone map is found and, finally, a simple gamut correction step is derived. Compared with the prior art, our rank-based method requires the fewest assumptions and delivers state-of-the-art radiometric calibration results.

References

  • [Anderson et al.(1995)Anderson, Motta, Chandrasekar, and Stokes] M. Anderson, R. Motta, S. Chandrasekar, and M. Stokes. Proposal for a standard default color space for the internet:srgb. In IS&T and SID’s 4th Color And Imaging Conference, pages 238–245. 1995.
  • [Barnard et al.(2002)Barnard, Martin, Coath, and Funt] Kobus Barnard, Lindsay Martin, Adam Coath, and Brian V. Funt. A comparison of computational color constancy algorithms. ii. experiments with image data. IEEE Transactions on Image Processing, 11(9):985–996, 2002.
  • [Berg et al.(2008)Berg, Cheong, Kreveld, and Overmars] Mark de Berg, Otfried Cheong, Marc van Kreveld, and Mark Overmars. Computational Geometry: Algorithms and Applications. Springer-Verlag TELOS, 3rd ed. edition, 2008.
  • [Chakrabarti et al.(2009)Chakrabarti, Scharstein, and Zickler] Ayan Chakrabarti, Daniel Scharstein, and Todd Zickler. An empirical camera model for internet color vision. In British machine Vision Conference, volume 1, page 4, 2009.
  • [Chakrabarti et al.(2014)Chakrabarti, Xiong, Sun, Darrell, Scharstein, Zickler, and Saenko] Ayan Chakrabarti, Ying Xiong, Baochen Sun, Trevor Darrell, Daniel Scharstein, Todd Zickler, and Kate Saenko. Modeling radiometric uncertainty for vision with tone-mapped color images. Transactions on pattern analysis and machine intelligence, 36(11):2185–2198, 2014.
  • [Debevec and Malik(1997)] Paul Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH, pages 369–378, 1997.
  • [Finlayson(1996)] G.D. Finlayson. Color in perspective. Transactions on Pattern analysis and Machine Intelligence, pages 1034–1038, October 1996.
  • [Finlayson et al.(2016)Finlayson, Darrodi, and Mackiewicz] Graham Finlayson, Maryam Mohammadzadeh Darrodi, and Michal Mackiewicz. Rank-based camera spectral sensitivity estimation. Journal of the Optical Society of America A, 33(4):589–599, 2016.
  • [Forsyth(1990)] David A Forsyth. A novel algorithm for color constancy. International Journal of Computer Vision, 5(1):5–35, 1990.
  • [Garcia et al.(2012)Garcia, Arora, and Gupta] Eric K. Garcia, Raman Arora, and Maya R. Gupta. Optimized regression for efficient function evaluation. IEEE Transactions on Image Processing, 21(9):4128–4140, 2012.
  • [Gevers et al.(2004)Gevers, Finlayson, and Schettini] Theo Gevers, Graham D. Finlayson, and Raimondo Schettini. Color for image indexing and retrieval. volume 94, pages 1–3, 2004.
  • [Giesen et al.(2007)Giesen, Schuberth, Simon, Zolliker, and Zweifel] J. Giesen, E. Schuberth, K. Simon, P. Zolliker, and O. Zweifel. Image-dependent gamut mapping as optimization problem. Image Processing, IEEE Transactions on, 16(10):2401–2410, Oct 2007.
  • [Gill et al.(1981)Gill, Murray, and Wright] Philip E. Gill, Walter Murray, and Margaret H. Wright. Practical optimization. Academic Press Inc. [Harcourt Brace Jovanovich Publishers], 1981.
  • [Gong and Cosker(2016)] Han Gong and Darren Cosker. Interactive removal and ground truth for difficult shadow scenes. Journal of the Optical Society of America A, 33(9):1798–1811, Sep 2016.
  • [Kim et al.(2012)Kim, Lin, Lu, Susstrunk, Lin, and Brown] Seon Joo Kim, Hai Ting Lin, Zheng Lu, Sabine Susstrunk, S. Lin, and M. S. Brown. A new in-camera imaging model for color computer vision and its application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(12):2289–2302, 2012. ISSN 0162-8828.
  • [Lin et al.(2012)Lin, Lu, Kim, and Brown] Hai Ting Lin, Zheng Lu, Seon Joo Kim, and Michael S. Brown. Nonuniform lattice regression for modeling the camera imaging pipeline. In European Conference on Computer Vision, volume 7572 of Lecture Notes in Computer Science, pages 556–568. Springer, 2012.
  • [Lin et al.(2004)Lin, Gu, Yamazaki, and Shum] S. Lin, Jinwei Gu, S. Yamazaki, and Heung-Yeung Shum. Radiometric calibration from a single image. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–938–II–945 Vol.2, June 2004.
  • [Petrov(1993)] A.P. Petrov. On obtaining shape from color shading. Color research and application, 18(4):236–240, 1993.
  • [Rakhmanov et al.(1994)Rakhmanov, Saff, and Zhou] E.A. Rakhmanov, E.B. Saff, and Y.M. Zhou. Minimal discrete energy on a sphere. Mathematical Research letters, pages 647–662, 1994.
  • [Ramanath et al.(2005)Ramanath, Snyder, Yoo, and Drew] R. Ramanath, W.E. Snyder, Y. Yoo, and M.S. Drew. Color image processing pipeline. Signal Processing Magazine, IEEE, 22(1):34–43, 2005.
  • [Reinhard et al.(2002)Reinhard, Stark, Shirley, and Ferwerda] Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. ACM Transaction on Graphics, 21(3):267–276, July 2002.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
33493
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description