LIME: A Method for Low-light IMage Enhancement

LIME: A Method for Low-light IMage Enhancement

Abstract

When one captures images in low-light conditions, the images often suffer from low visibility. This poor quality may significantly degrade the performance of many computer vision and multimedia algorithms that are primarily designed for high-quality inputs. In this paper, we propose a very simple and effective method, named as LIME, to enhance low-light images. More concretely, the illumination of each pixel is first estimated individually by finding the maximum value in R, G and B channels. Further, we refine the initial illumination map by imposing a structure prior on it, as the final illumination map. Having the well-constructed illumination map, the enhancement can be achieved accordingly. Experiments on a number of challenging real-world low-light images are present to reveal the efficacy of our LIME and show its superiority over several state-of-the-arts.

1Introduction

High-visibility images reflect clear details of target scenes, which are critical to many vision-based techniques, such as object detection [1] and tracking [2]. But, images captured in low-light conditions are often of low visibility. Besides degrading the visual quality of images, it very likely hurts the performance of algorithms that are primarily designed for high-visibility inputs. Figure 1 provides three such examples, from which, we can see that many details, the paintings on the wall in the first case for example, have almost been “buried” in the dark. To make the buried information visible again, low-light image enhancement is demanded.

Figure 1: Upper Row: Three natural low-light images. Lower Row: The enhanced results by our method.
Figure 1: Upper Row: Three natural low-light images. Lower Row: The enhanced results by our method.

Directly amplifying the low-light image is probably the most intuitive and simplest way to recall the visibility of dark regions. But this operation gives birth to another problem, say relatively bright regions might be saturated and thus loss corresponding details. Histogram equalization strategies [3] can avoid the above problem by somehow forcing the output image to fall in the range . However, in nature, they focus on contrast enhancement instead of exploiting real illumination causes, having the risk of over- and under-enhancement. The method proposed in [6] tries to enhance contrast while preserving naturalness of illumination. Although it prevents the result from over-enhancement, in our test, its performance is not so attractive in both efficiency and visual quality.

As noticed in [7], inverted low-light images look like hazy images, as shown in Figure 2. Based on this observation, the authors of [7] alternatively resorted to dehaze the inverted low-light images. After dehazing, the obtained unrealistic images is inverted again as the final enhanced results. Recently, Li et al. followed this technical line and further improved the visual quality by first over-segmenting the input image and then adaptively denoising different segments [8]. Even though the above methods can provide reasonable results, the basic model they rely on is lacking in physical explanation. This paper will try to connect this un-rooted model to a more physically meaningful one that our method adopts.

Figure 2: The inverted versions (unrealistic images) of those shown in the upper row of Fig. .
Figure 2: The inverted versions (unrealistic images) of those shown in the upper row of Fig. .

This work intends to enhance a low-light image by estimating its illumination map. The illumination map is first constructed by finding the maximum intensity of each pixel in R, G and B channels. Then, we exploit the structure of the illumination and execute structure-aware smoothing to refine the illumination map. Experiments on a number of challenging images are conducted to demonstrate the advantages of our method in comparison with other state-of-the-art methods.

2Proposed Method

Our method is built upon the following model, which explains the formation of a low-light image:

where and are the captured image and the desired recovery, respectively. In addition, represents the illumination map, and the operator means element-wise multiplication. The model is with clear physical meaning, say the observed image can be decomposed into the product of the desired scene and the illumination image. As can be seen, the estimation of is key to the recovery of .

As mentioned, another widely used model is based on the observation that inverted low-light images look similar to haze images, which is thus expressed as [9]:

where represents the global atmospheric light. Although the visual effect of inverted low-light images is intuitively similar to haze images, compared to the model , the physical meaning of the above is not easy to directly explain. We will show the relation between and later.

2.1Initial Illumination Map Estimation

As one of the first color constancy methods, Max-RGB [12] tries to estimate the illumination by seeking the maximum value of three color channels, say R, G and B. But this estimation can only boost the global illumination. In this paper, to handle non-uniform illuminations, we alternatively adopt the following initial estimation:

The obtained guarantees that the recovery will not be saturated, because of

where is a very small constant to avoid the zero denominator. Let us here recall the dark channel prior, a commonly used prior to estimate the transmission map for dehazing [9], on as follows:

Accordingly, substituting into yields:

We can see that when , both and reach the same result. But, if gets away from , the equivalence between the model [7] and breaks (see Figure 3 for difference).

Figure 3: Comparison of  and  with the same illumination map. The atmospheric light a estimated by  is larger than 0.95. Even though, the difference is still noticeable.
Figure 3: Comparison of and with the same illumination map. The atmospheric light estimated by is larger than . Even though, the difference is still noticeable.

In this work, we employ to initially estimate illumination map , due to its simplicity, although various approaches, like [13], have been developed to improve the accuracy in past decades. Most of these improvements essentially consider the local consistency of illumination by taking into account neighboring pixels within a small region around the target pixel. In the following, we provide a more powerful scheme to better achieve this goal.

Figure 4: Comparison of different illumination maps and corresponding enhanced results. From (a) to (f): Illumination map estimated individually on each pixel (Initial), refined by local max  (Max), bilateral filtering (BF), \ell_2 loss total variation minimization (TV), and our structure-aware smoothing, respectively.
Figure 4: Comparison of different illumination maps and corresponding enhanced results. From (a) to (f): Illumination map estimated individually on each pixel (Initial), refined by local max (Max), bilateral filtering (BF), loss total variation minimization (TV), and our structure-aware smoothing, respectively.

2.2Illumination Map Refinement

As aforementioned, the illumination estimation can benefit from local consistency. Two representative ways are:

where is a region centered at pixel , and is the location index within the region. These strategies can somewhat enhance the local consistency, but they are structure-blind.

A “good” solution should simultaneously preserve the overall structure and smooth the textural details. To address this issue, based on the initial illumination map , we propose to solve the following optimization problem:

where ( for all the experiments) is the coefficient to balance the involved two terms and, and designate the Frobenious and norms, respectively. Further, is the weight matrix, and that contains (horizontal) and (vertical), is the first order derivative filter. In the objective , the first term takes care of the fidelity between the initial map and the refined one , while the second term considers the (structure-aware) smoothness. It can be seen that setting the weight matrix to (all entries being ) leads to a classic loss total variation minimization problem (TV for short) [16], which is also short of ability to distinguish between strong structural edges and texture [17].

Hence, the key is the design of . Inspired by RTV [17], for each location, the weight (e.g. ) is set via:

where is produced by the Gaussian kernel with the standard deviation (we use throughout this paper), and is the absolute value operator. Please note that, different to RTV, our weight matrix is constructed based on the given instead of being iteratively updated according to . That means only needs to be calculated once.

Traditionally, the problem can be effectively solved via alternating direction minimization techniques. To speed up the calculation, we approximate by the following:

As can be seen, the problem now only involves quadratic terms. Thus, the solution can be directly computed without requiring any iterations. Figure 4 shows a comparison of different methods on illumination map, from which, we can see the advance of our method.

2.3Other Operations

Figure 5: Comparison of different methods without denoising involved.
Comparison of different methods without denoising involved.
Figure 5: Comparison of different methods without denoising involved.
Figure 6: Gamma correction, denosing and recomposition. (a)-(c) are the recovered images using \mathbf{T}^{\gamma} with \gamma=0.5, \gamma=0.8 and \gamma=1, respectively. The corresponding illumination map is given in the up-right corner of each sub-picture. Noises appear in the enhanced images. (d) is the denoised version of (b) by BM3D, while (e) is the recomposed result of (b) and (e) by . It can be seen from the zoomed-in patches that the recomposition adaptively keeps the fine details of the bright region and suppresses the noises of the dark region.
Figure 6: Gamma correction, denosing and recomposition. (a)-(c) are the recovered images using with , and , respectively. The corresponding illumination map is given in the up-right corner of each sub-picture. Noises appear in the enhanced images. (d) is the denoised version of (b) by BM3D, while (e) is the recomposed result of (b) and (e) by . It can be seen from the zoomed-in patches that the recomposition adaptively keeps the fine details of the bright region and suppresses the noises of the dark region.

Having the refined illumination map , we can recover by following . One can also manipulate the illumination map through gamma transformation, say . From the upper row of Figure 6, we can see the difference between the results by setting to , and . For the rest experiments, we adopt . Moreover, possible noises previously hiding in the dark are also accordingly amplified, especially for the very low-light inputs (regions), as shown in Figure 6. Denoising techniques are required to further improve the visual quality. Many off-the-shelf denosing tools, such as [18], can be employed to do the job. Considering the comprehensive performance, BM3D [18] is the choice of this work. In our implementation, for further cutting the computational load, we only execute BM3D on the Y channel by converting from the RGB colorspace into the YCbCr one. In addition, the magnitude of noises is not the same for different regions of the input, as the amplification is different. And BM3D treats different patches equally. Therefore, to avoid the unbalance of processing, e.g. some (dark) places are well-denoised while some (brighter) over-smoothed, we employ the following operation:

where and are the results after denoising and recomposing, respectively. The merit of this operation can be viewed from Figure 6 (e), compared with Figure 6 (d). We mention that the denoising, as a post-processing step, can be concatenated to any low-light image enhancing method.

Figure 7: Comparison of results without and with denoising.
Figure 7: Comparison of results without and with denoising.

3Experimental Results

In this section, we compare our LIME with several state-of-the-art methods, including histogram equalization (HE), adaptive histogram equalization (AHE), Gamma Correction (GC), Dehazing based method [7] (DeHz) and Naturalness Preserved Enhancement algorithm (NPE) [6]. All the codes are in Matlab1, which ensures the fairness of time comparison. All the experiments are conducted on a machine running Windows 7 OS with 64G RAM and 2.4GHz CPU.

Figure 5 provides several comparisons. From the top row (the input is the second case of Figure 1 with size x), we can see that AHE can not effectively recall the information in dark regions while GC () changes the color of the whole image. These problems almost exist always, therefore we discard them for the rest comparisons. HE, DeHz and NPE outperform AHE and GC in this case, but are inferior to our method in terms of visual quality. In time cost, although LIME spends more than HE, AHE and GC, it is comparable to or even more efficient than DeHz, while much more efficient than NPE. Most cost of DeHz comes from the estimation of atmospheric light. Two more comparisons are given in Figure 5, which are the additional evidence of the advantage of LIME, compared with HE, DeHz and NPE.

Figure 7 gives another test. The very low-light input hides intensive noises in the dark. After performing LIME, the details of the scene get enhanced, but the noises also come out, as shown in the middle of Figure 7. This is an inevitable problem encountered by almost all of existing low-light enhancement algorithms. As we have discussed in Section 2.3, denoising is required. The right picture in Figure 7 is the denoised result by executing BM3D on the middle of Figure 7, from which we can see the improvement in terms of visual quality. To allow more experimental verification and comparisons, we provide our code at http://cs.tju.edu.cn/orgs/vision/~xguo/homepage.htm

4Conclusion

This paper has proposed an efficient and effective method to enhance low-light images for boosting the visual quality and offering contemporary vision applications with reliable inputs. The key to the enhancement is how well the illumination map is estimated. The structure-aware smoothing has been developed to improve the illumination consistency. The experimental results have revealed the advance of our method compared with several state-of-the-art alternatives.

Footnotes

  1. HE and AHE uses histeq and adapthisteq functions integrated in the Matlab toolbox. GC is achieved by , while the code of NPE is downloaded from the authors’ website. The code of DeHz is not publicly available when this paper is prepared, but it is easy to be implemented based on [9].

References

  1. D. Oneata, J. Revaud, J. Verbeek, and C. Schmid, “Spatio-temporal object detection proposals,” in Proceedings of European Conference on Computer Vision (ECCV), pp. 737–752, 2014.
  2. K. Zhang, L. Zhang, and M. Yang, “Real-time compressive tracking,” in Proceedings of European Conference on Computer Vision (ECCV), pp. 866–879, 2014.
  3. E. Pisano, S. Zong, B. Hemminger, M. DeLuce, J. Maria, E. Johnston, K. Muller, P. Braeuning, and S. Pizer, “Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms,” Journal of Digital Imaging, vol. 11, no. 4, pp. 193–200, 1998.
  4. H. Cheng and X. Shi, “A simple and effective histogram equalization approach to image enhancement,” Digital Signal Processing, vol. 14, no. 2, pp. 158–170, 2004.
  5. M. Abdullah-Al-Wadud, M. Kabir, M. Dewan, and O. Chae, “A dynamic histograme equalization for image contrast enhancement,” IEEE Trans. on Consumer Electronics, vol. 53, no. 2, pp. 593–600, 2007.
  6. S. Wang, J. Zheng, H. Hu, and B. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” IEEE Trans. on Image Processing (TIP), vol. 22, no. 9, pp. 3538–3578, 2013.
  7. X. Dong, G. Wang, Y. Pang, W. Li, J. Wen, W. Meng, and Y. Lu, “Fast efficient algorithm for enhancement of low lighting video,” in Proceedings of IEEE Conference on Multimedia & Expo (ICME), pp. 1–6, 2011.
  8. L. Li, R. Wang, W. Wang, and W. Gao, “A low-light image enhancement method for both denoising and contrast enlarging,” in Proceedings of IEEE Conference on Image Processing (ICIP), pp. 3730–3734, 2015.
  9. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), vol. 33, no. 12, pp. 2341–2353, 2011.
  10. S. Narasimhan and S. Nayar, “Contrast restoration of weather degraded images,” IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), vol. 25, no. 6, pp. 713–724, 2003.
  11. G. Meng, Y. Wang, J. Duan, S. Xiang, and C. Pan, “Efficient image dehazing with boundary constraint and contextual regularization,” in Proceedings of International Conference on Computer Vision (ICCV), pp. 617–624, 2013.
  12. E. Land, “The retinex theory of color vision,” Scientific American, vol. 237, no. 6, pp. 108–128, 1977.
  13. D. Forsyth, “A novel approach to color constancy,” in Proceedings of International Conference on Computer Vision, pp. 9–18, 1988.
  14. B. Funt and L. Shi, “The rehabilitation of maxrgb,” in Proceedings of Color Imaging Conference, pp. 256–259, 2010.
  15. H. Joze, M. Drew, G. Finlayson, and P. Rey, “The role of bright pixels in illumination estimation,” in Proceedings of European Conference on Colour in Graphics, Imaging and Vision, 2012.
  16. S. Chan, R. Khoshabeh, K. Gibson, P. Gill, and T. Nguyen, “An augmented lagrangian method for total variation video restoration,” IEEE Trans. on Image Processing (TIP), vol. 20, no. 11, pp. 3097–3111, 2011.
  17. L. Xu, Q. Yan, Y. Xia, and J. Jia, “Strucutre extraction from texture via relative total variation,” ACM Trans. on Graphics (TOG), vol. 31, no. 6, pp. 139:1–139:10, 2013.
  18. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3d transform-domain collaborative filtering,” IEEE Trans. on Image Processing (TIP), vol. 16, no. 8, pp. 2080–2095, 2007.
  19. H. Burger, C. Schuler, and S. Harmeling, “Image denoising: Can plain neural networks compete with bm3d,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2392–2399, 2012.
  20. S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with applications to image denoising,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2862–2869, 2014.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minumum 40 characters
Add comment
Cancel
Loading ...
13133
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description