Denoising Imaging Polarimetry by Adapted BM3D Method

Denoising Imaging Polarimetry by Adapted BM3D Method

Alexander B. Tibbs Ilse M. Daly School of Biological Sciences, University of Bristol, BS8 1TQ Nicholas W. Roberts School of Biological Sciences, University of Bristol, BS8 1TQ David R. Bull Department of Electrical and Electronic Engineering, University of Bristol, BS8 1UB
Abstract

Imaging polarimetry allows more information to be extracted from a scene than conventional intensity or colour imaging. However, a major challenge of imaging polarimetry is image degradation due to noise. This paper investigates the mitigation of noise through denoising algorithms and compares existing denoising algorithms with a new method, based on BM3D. This algorithm, PBM3D, gives visual quality superior to the state of the art across all images and noise standard deviations tested. We show that denoising polarization images using PBM3D allows the degree of polarization to be more accurately calculated by comparing it to spectroscopy methods.

\journal

josaa \datesCompiled June 28, 2019 \doi

1 Introduction

The polarization of light describes how light waves propagate through space [1]. Although different forms of polarization can occur, such as circular polarization, in this paper we focus only on linear polarization, the form that is abundant in nature. Light is said to be completely linearly polarized (or polarized for the purposes of this paper) when all waves travelling along the same path through space are oscillating within the same plane. If however, there is no correlation between the orientation of the waves, the light is described as unpolarized. Polarized and unpolarized light are the limiting cases of partially polarized light, which can be considered to be a mixture of polarized and unpolarized light.

The polarization of light can be altered by the processes of scattering and reflection. As a form of visual information it provides a fitness benefit such that many animals [2, 3] use polarization sensitivity for a variety of task-specific behaviours such as: navigation [4], communication [5] and contrast enhancement [6]. Inspired by nature, there are now many devices that capture images containing information about the polarization of light [7, 8], known as imaging polarimeters or polarization cameras. These have been used in a growing number of applications [9] including mine detection [10], surveillance [11], shape retrieval [12] and robot vision [13], as well as research in sensory biology [8, 14].

A major challenge facing imaging polarimetry, addressed in this paper, is noise. State of the art imaging polarimeters suffer from high levels of noise, and it will be shown that conventional image denoising algorithms are not well suited to polarization imagery.

Whilst a great deal of previous work has been done on denoising, very little has specifically been tailored to imaging polarimetry. Zhao et al. [15] approached denoising imaging polarimetry by computing Stokes components from a noisy camera using spatially adaptive wavelet image fusion, whereas Faisan et al’s [16] method is based on a modified version of the nonlocal means (NLM) algorithm [17].

This paper compares the effectiveness of conventional denoising algorithms in the context of imaging polarimetry. A novel method termed PBM3D, adapted from an existing denoising algorithm, BM3D, is then introduced and will be shown to be superior to the state of the art. The use of appropriate test imagery will also be discussed.

2 Imaging polarimetry

2.1 Representing light polarization

A polarizer is an optical filter which only transmits light of a given linear polarization. The angle between the transmitted light and the horizontal is known as the polarizer orientation. Let represent the total light intensity and represent the intensity of light which is transmitted through a polarizer orientated at degrees to the horizontal. The standard way of representing the linear polarization is by using Stokes parameters  [18], which are defined as follows:

(1)
(2)
(3)

Note that , so the above can be rewritten, using , and as:

(4)
(5)
(6)

The degree of (linear) polarization () and the angle of polarization () are defined as:

(7)
(8)

The represents the proportion of light that is polarized, rather than being unpolarized, i.e. means that the light is fully polarized, means unpolarized. The represents the average orientation of the oscillation of multiple waves of light, expressed as an angle from the horizontal.

2.2 Imaging polarimeters

Imaging polarimeters are devices which, in addition to measuring the intensity of light at each pixel in an array, also measure the polarization of light at each pixel location. There are many designs of imaging polarimeter, summarised in [19]. The common feature they share is measuring the intensity of light which passes through polarizers of multiple orientations, , possibly with additional measurements of circular polarization, at each pixel in an array. The measurements for multiple orientations are taken either simultaneously or of a completely static scene. The Stokes parameters are then derived at each pixel. For the rest of this paper we will consider a polarimeter that measures , , and , as this is the most common type [8, 7]. Generalisations to imaging polarimeters which measure intensities at different angles are straightforward.

As this paper addresses polarization measurements across an array, the symbols , , , , , , and will henceforth refer to the array of values, rather than a single measurement. , and are known as the camera components, and , and as the Stokes components.

2.3 Noise

Noise affects most imaging systems and is especially challenging in polarimetry due to the complex sensor configuration involved with measuring the polarization. Each type of imaging polarimeter (see [19] for a description of the different types) is affected by noise to a greater extend than conventional cameras are. ‘Division of focal plane’ polarimeters, which use micro-optical arrays of polarization elements, suffer from imperfect fabrication and crosstalk between polarization elements. ‘Division of amplitude’ polarimeters, which split the incident light into multiple optical paths, suffer from low SNR due to the splitting of the light. ‘Division of aperture’ polarimeters, which use separate apertures for separate polarization components, suffer from distortions due to parallax. ‘Division of time’ polarimeters require static scenes, and are incapable of recording video, so for many applications cannot be used. Also, in polarimetry, where and are often the quantities of interest, they are nonlinear functions of the camera and Stokes components and this has the effect of amplifying the noise degradation.

To highlight the degradation of a image due to noise consider figure 1. The top row shows the three camera components of an unpolarized scene (i.e. all three components are identical, and everywhere). The original images with noise added are shown in the bottom row. Although there is only a small noticeable difference between the original and noisy camera components, the difference between the original and noisy images is severe. This indicates a large error, with 25% of pixels exhibiting error greater than 10%. The error is greater where the intensity of the camera components is smaller. To see why this is the case consider a noisy Stokes image , where the measured values are normally distributed around the true Stokes parameters . Let the true be given by . The naive way to compute is to apply the formula to the measured Stokes parameters . But (where is expected value) meaning that this is a biased estimator, so the calculated does not average to the correct result. This can be seen by the fact that if the true , , is zero, then any error in and results in . Denoising algorithms, including the one proposed in this paper, PBM3D, are thus essential for mitigating such degradations due to noise.

Figure 1: Simulation of an unpolarized scene with and without noise (). Black represents a value of 0, white of 1. The error is large for the noisy image.

This paper considers only Gaussian noise, as this is typical of that which affects imaging polarimeters. A noisy camera component, , is described as follows. Let denote the image domain. For all and where the noise, , is a normally distributed zero-mean random variable with standard deviation , and is the true camera component.

3 State of the art

3.1 Polarization denoising algorithms

There are various methods for mitigating noise degradation in imaging polarimetry. For example, polarizer orientations can be chosen optimally for noise reduction [20, 21], however this is not always possible due to constraints on the imaging system. Further reductions in noise can also be made through the use of denoising algorithms, which attempt to estimate the original image.

While there exists a vast literature on denoising algorithms in general, little is specifically targeted at denoising imaging polarimetry. Zhao et al. [15] approached denoising imaging polarimetry by computing Stokes components from a noisy camera using spatially adaptive wavelet image fusion, based on [22]. A benefit of this algorithm is that the noisy camera components need not be registered prior to denoising. The algorithm of Faisan et al. [16] is based on a modified version of Buades et al.’s nonlocal means (NLM) algorithm [17]. The NLM algorithm is modified by reducing the contribution of outlier patches in the weighted average, and by taking into account the constraints arising from the Stokes components having to be mutually compatible. A disadvantage of this method is that it takes a long time to denoise a single image (550s for a image which takes approximately 1s using our method, PBM3D. Both on an Intel Core i7, running at 3 GHz).

In this paper, our PBM3D algorithm will be compared to the above two algorithms. Faisan et al. [16] compared their denoising algorithm with earlier methods [23, 24, 25] and demonstrated that their NLM based algorithm gives superior denoising performance. For this reason, comparison to these algorithms is not considered.

3.2 Test imagery

In order to demonstrate the effectiveness of denoising algorithms, they must be evaluated using representative noisy test imagery. There are three types of test imagery used in the literature. The first is genuine polarization imagery captured by imaging polarimeters. This is ultimately what will be denoised, but it only allows for qualitative visual assessment as there is no ground truth, i.e. the original noise-free image is not available. This means that PSNR (peak signal to noise ratio) or other reference-based image quality metrics cannot be calculated. In order to have a ground truth for quantitative analysis, both the noisy and noise-free versions of the same image must be available.

The second type of test imagery comprises synthetic images with simulated noise. Synthetic images have the advantage of being completely controllable, so the effect of varying any parameter on denoising performance can be investigated. The disadvantage of using synthetic images is that they may not be fully representative of natural images, which can lead to algorithms appearing more or less effective than they would be with real images, especially if the synthetic test images are overly simplified. The simulated noise added may also not accurately reflect the properties of the real noise.

The third type of test imagery comprises noise-free polarization images, with simulated noise. Although it is not possible to produce noise-free images with a noisy imaging polarimeter, there are approaches which allow the noise level to be reduced to arbitrarily small levels; this is discussed in section 5. The advantage of this type of test imagery is that the images are natural, so are likely, and can be chosen to be, similar to what will ultimately be denoised, which is dependent on the final application. Using simulated noise, however, does mean that the noise properties may be unrealistic. Of the imaging polarimetry denoising papers, only Valenzuela and Fessler [24] used real polarization images with simulated noise. Sfikas et al., Zallat & Heinrich, and Faisan et al. [23, 25, 16] used simple synthetic images, consisting of simple geometric shapes with regions of uniform, or smoothly varying, Stokes components. Such images are easy to denoise using basic uniform or smoothly varying regions, and as such they don’t test the algorithms’ ability to denoise natural images. Faisan et al. [15] used only real polarimetric images, and as such no PSNR values were given, making quantitative analysis difficult.

4 Method

Our approach to denoising polarization images is to adapt Dabov’s BM3D algorithm [26] for use with imaging polarimetry, a novel method which we call PBM3D.

BM3D was chosen primarily for its robustness and effectiveness. Sadreazami et al. [27] recently compared the performance of a large number of state of the art denoising algorithms, using three test images and four values of , the noise standard deviation. They showed that no one denoising algorithm of those tested always gave the greatest denoised PSNR. However BM3D was always able to give denoised PSNR values close to the best performing algorithm, and in more than half the cases was in the top two. Another appealing aspect of BM3D is that extensions have been published for color images (CBM3D) [28], multispectral images (MSPCA-BM3D) [29], volumetric data (BM4D) [30] and video (VBM4D) [31]. This extensibility shows the versatility of the core algorithm. Sadreazami et al. found that CBM3D was the best performing algorithm for color images with high noise levels.

4.1 Bm3d

BM3D consists of two steps. In step 1 a basic estimate of the denoised image is produced, step 2 then refines the estimate produced in step 1 to give the final estimate. Steps 1 and 2 both consist of the same basic substeps, show in algorithm 1.

1:for each block (rectangular neighbourhood of pixels) in noisy image do
2:     find similar blocks across the image for step 1 this is done using the noisy image, for step 2 the basic estimate
3:     stack similar blocks to form 3D group
4:     apply 3D transform to obtain sparse representation
5:     apply filter to denoise for step 1 the filtering is done using a hard thresholding operation, and for step 2 a Wiener filter is used
6:     invert transform
7:for each pixel do
8:     estimate single denoised value from values of multiple overlapping blocks
9:return denoised image
Algorithm 1 BM3D - single step

BM3D is described more fully in [26], and thorough analysis is provided in [32].

4.2 Cbm3d

Figure 2: Basic outline of the CBM3D/PBM3D denoising algorithm.

CBM3D adapts BM3D for colour images [28]. Figure 2 outlines the algorithm, which works by applying BM3D to the three channels of the image in the color space, also in two steps, but computing the groups only using the channel. The details of CMB3D are as shown in algorithm 2.

1:input noisy colour image
2:apply colour-space transform represents a chosen luminance-chrominance colour space
3:for each block in channel image do
4:     find similar blocks across the image for step 1 this is done using the noisy image, for step 2 the basic estimate
5:      stack similar blocks to form 3D group
6:for channels  do
7:     stack blocks to form 3D groups using the same groups as formed in line 5
8:for each channel  do
9:     for each group do
10:         apply 3D transform to obtain sparse representation
11:         apply filter to denoise for step 1 the filtering is done using a hard thresholding operation, and for step 2 a Wiener filter is used
12:         invert transform      
13:     for every pixel do
14:         estimate single denoised value from values of multiple overlapping blocks      
15:Apply inverse colour-space transform
16:return denoised image
Algorithm 2 CBM3D - single step

Dabov et al [28] provide the following reason for why CBM3D performs better than applying BM3D separately to three colour channels:

  • The SNR of the intensity channel, is greater than the chrominance channels.

  • Most of the valuable information, such as edges, shades, objects and texture are contained in .

  • The information in and is tends to be low-frequency.

  • Isoluminant regions, with and varying are rare.

If BM3D is performed separately on colour channels, and , the grouping suffers [28] due to the lower SNR, and the denoising performance is worse as it is sensitive to the grouping.

4.3 Pbm3d

Colour imagery and polarization imagery are analogous in the way demonstrated in figure 3, where color and polarization components of the same scene are shown. The , , and images, like , , are visually similar to one another, each containing most of the information about the scene with small differences due to the colour in the case of and polarization in the case of . When is transformed into , most of the information is contained in the luminance, . In much the same way, when the polarization components are transformed from into using the standard Stokes transformation, most of the information is contained in .

Figure 3: Scene imaged in colour and in polarization. The top row shows the components of a colour image. The second row shows the same image in the color space. The third row shows the same scene as three polarization components . The bottom row shows polarization image transformed into Stokes parameters. Photo credit: Martin How

In order to optimise BM3D for polarization images, we propose taking CBM3D and replacing the to transformation with a transformation from the camera component image image to a chosen luminance-polarization space, denoted generally as , as depicted in figure 2. This is shown in algorithm 3.

1:input noisy polarization image
2:apply polarization transform represents a chosen luminance-polarization colour space
3:for each block in channel image do
4:     find similar blocks across the image for step 1 this is done using the noisy image, for step 2 the basic estimate
5:      stack similar blocks to form 3D group
6:for channels  do
7:     stack blocks to form 3D groups using the same groups as formed in line 5
8:for each channel  do
9:     for each group do
10:         apply 3D transform to obtain sparse representation
11:         apply filter to denoise for step 1 the filtering is done using a hard thresholding operation, and for step 2 a Wiener filter is used
12:         invert transform      
13:     for every pixel do
14:         estimate single denoised value from values of multiple overlapping blocks      
15:Apply inverse colour-space transform
16:return denoised image
Algorithm 3 PBM3D - single step

The motivation for denoising polarimetry in this manner is the same as for colour images. The polarization parameters are analogous to the components in the following ways:

  • , like , tends to have higher SNR than and in natural polarization images.

  • , like , contains most of the valuable information.

  • Regions with constant and varying and are uncommon.

4.4 Choice of polarization transformation

PBM3D relies on a linear transformation, represented by matrix , to convert camera components into intensity-polarization components, the choice of has a large effect on denoising performance. Which matrix is optimal is dependent on the image statistics and the noise level, which are both dependent on the application. Here we describe how to estimate the optimal matrix, , given a set of noise-free model images, , and a given noise standard deviation, .

Let be a noise free camera component image (e.g. ), be with Gaussian noise of standard deviation added, be the set of images and represent the operation of applying PBM3D with transformation matrix . Define as follows:

(9)

where is the mean square error. Note that is normalised such that for each row , .

Due to the large number of degrees of freedom of , and the fact that the matrix elements can take any value in the range , it is not possible to perform an exhaustive search. Instead a Monte Carlo method, and a pattern search method can be used and are described here. Results from using both methods are shown in section 5.

4.4.1 Monte Carlo method

This method has the advantages of being simple to implement, and not susceptible to convergence to local minima, but has the disadvantage of being slow to converge. It is shown in algorithm 4.

1:for the desired number of rounds do
2:     randomly choose a valid matrix
3:     for each image  do
4:         denoise using
5:         compute the mean MSE between every denoised image and its original      
6:return , the matrix such that the mean MSE is smallest
Algorithm 4 Monte Carlo method

4.4.2 Pattern search method

This method has faster convergence, but can converge to local rather than global minima. It is shown in algorithm 5. Note that the intervals and are both used to avoid converging to non-global minima.

1:choose a starting matrix and small interval
2:
3:loop
4:     find all perturbations of by which preserve the normalisation condition for each row ,
5:     find all perturbations of by which preserve the normalisation condition
6:     for every perturbation, , of  do
7:         for every image,  do
8:              denoise using          
9:               
10:     
11:     if  then return      
12:     
Algorithm 5 Pattern search method

5 Experiments

5.1 Datasets

As noise-free polarization images cannot be produced using a noisy imaging polarimeter, we instead use a DSLR camera with a rotatable polarizer in front of the lens. This approach to producing imaging polarimetry is one of the earliest [33] and has been used by various authors e.g. [6, 34].

For this approach to work, the camera sensor must have a linear response with respect to intensity, that is , where and are the measured and actual light intensities, and is an arbitrary constant. The linearity can be verified using a fixed light source and a second rotating polarizer. As one polarizer is kept stationary, and the other is rotated, the intensity values measured at each pixel will produce a cosine squared curve if the sensor is linear, according to Malus’ law [18]. The DSLR used to generate the images in this paper was a Nikon D70, whose sensors have a linear response.

The images are generated as follows:

1:All camera settings are set to manual for consistency between shots
2:To prevent inaccuracies due to compression the camera is set to take images in raw format
3:The camera is placed on a tripod or otherwise such that it is stationary
4:The polarizer is orientated to be parallel to the horizontal and an image is taken
5:The polarizer is rotated so that it is at to the horizontal and a second image is taken
6:The polarizer is rotated so that it is orientated vertically and the final image is taken

Given the superior SNR of modern DSLR cameras, this provides low noise polarization images. For arbitrarily low noise levels, multiple photos for a given polarizer angle are taken, registered and averaged. The main drawback of this method is that the light conditions and image subjects must be stationary, this method therefore cannot be used for many applications, but still allows noise free polarization images to be taken, so is invaluable for testing denoising algorithms.

5.2 Optimal denoising matrix

The optimal matrix for a given application is dependent on the image statistics and the noise level. In order to test the matrix optimisation algorithms given in Section 4, and with no particular application in mind we produced a set of 10 polarization images, using the method above, of various outdoor scenes. We added noise of several values of , the noise standard deviation (see tables 1 & 2). The optimal matrices given in this section are therefore only optimal for this particular image set. However they provide a useful starting point and are likely to be close to optimal for applications where the images involve outdoor scenes. The choice of 10 images was arbitrary. Using a larger number of images would result in a more robust estimate of the optimal matrix. The component of each image is shown in figure 4.

Figure 4: of each image in the set.

The natural choice of transform to gain an intensity-polarization representation of a polarization image from the camera components is to use a Stokes transformation, which, after normalization is given by:

(10)

However it was found during experiments that the opponent transform, matrix of CBM3D [28], given below, almost always gives superior denoising performance to the Stokes transform.

(11)

This is logical because taking the mean of the three camera components gives a greater SNR than taking the mean of only two components, and having greater SNR gives better grouping in the PBM3D algorithm, which is important as denoising performance is very sensitive to the quality of the grouping. The opponent matrix was therefore taken as the initial matrix, in the pattern search algorithm.

Both methods were applied to the model imagery at four values. The Monte Carlo method was performed for 4000 rounds. The pattern search method was applied with . Table 1 shows the PSNR values for images denoised using the estimated optimal matrices. It can be seen that in almost every case the matrix found using the pattern search method results in the most effective denoising.

street dome building
I S O M P I S O M P I S O M P
0.01 32.6 32.6 32.6 46.0 46.2 44.8 45.7 46.5 46.6 46.8 40.9 41.2 41.5 46.6 47.0
0.057 30.9 31.1 31.3 36.1 36.3 37.5 38.5 39.2 39.1 39.2 35.0 35.8 36.5 37.2 37.3
0.1 28.7 29.2 29.4 31.7 31.8 32.9 33.1 33.5 33.8 33.6 31.0 32.5 33.3 33.4 33.6
0.15 26.7 26.8 27.1 28.2 28.3 29.7 29.0 29.1 30.2 32.0 28.7 29.4 30.2 30.2 30.3
Table 1: PSNR values for images (street, dome, building) denoised using the following matrices: I: identity matrix; S: Stokes matrix; O: opponent matrix; M: Monte Carlo optimal; P: pattern search optimal. is the noise standard deviation. Bold indicates maximum PSNR.

The pattern search method was then applied at 10 sigma values, giving an estimated optimal matrix for each (table  2).

optimal matrix
0.01
0.026
0.041
0.057
0.072
0.088
0.1
0.12
0.13
0.15
Table 2: The optimal matrices computed using the pattern search method for 10 values of , the noise standard deviation.

The pattern search method was also applied to an image set containing all 10 images, each with noise added of 10 different values. The following matrix was found to be optimal on average across all values:

(12)

5.3 Comparison of denoising algorithms

The performance of PBM3D with a variety of images (different to those used for the matrix optimization) and noise levels was compared to the performance of several other denoising algorithms for polarization:

  • BM3D: Standard BM3D for grayscale images applied individually to each camera component

  • BM3D Stokes: Standard BM3D applied individually to each Stokes component , found by transforming the camera components.

  • Zhao: Zhao et al’s method [15]

  • Faisan: Faisan et al’s method [16]

In order to quantitatively compare the denoising performance PSNR was computed for each denoised image.

For Stokes image with ground truth given by , with , , and , where is the image domain, PSNR is given by

(13)

where

(14)

Table 3 shows the PSNR values for four images (‘oranges’, ‘cars’, ‘windows’, ‘statue’). The same data, along with that for four other images is plotted in figure 5. It can be seen that PBM3D always provided the greatest denoising performance. Every image denoised using PBM3D at every noise level had a greater PSNR than images denoised using all other methods. The second best performing method in every case was BM3D Stokes, with PBM3D denoising images with a greater PSNR of 0.84dB on average. The difference in PSNR between images denoised using PBM3D and BM3D Stokes was greatest for the intermediate noise levels. The smaller noise levels exhibited less of a difference, and the PSNR values in the higher noise values became closer as noise was increased.The convergence of the PSNR values in the higher noise levels can be explained by the fact that the and components of the images become so noisy that there they bear little resemblance to the ground truth, as can be seen in figure 9. Zhao’s method performed poorly at all noise levels, it provided a smaller PSNR at higher noise levels than the other methods at higher noise levels. Faisan’s method had worse performance than all of the BM3D-based methods, at all noise levels (images denoised using Faisan had a PSNR 4.50dB smaller on average than those denoised using PBM3D), but performed significantly better than Zhao’s method.

oranges cars windows statue
B S P Z F B S P Z F B S P Z F B S P Z F
0.010 47.3 48.3 49.0 34.7 45.6 45.6 46.4 47.0 28.4 43.0 44.8 46.1 47.1 24.5 42.1 45.2 46.5 47.3 26.9 42.5
0.026 43.4 44.2 44.9 34.7 41.8 40.4 41.2 41.9 28.4 37.5 39.1 40.5 41.5 24.5 35.4 40.1 41.4 42.1 26.9 36.4
0.041 41.5 42.3 43.0 34.6 39.9 38.0 38.9 39.5 28.4 35.2 36.3 37.7 38.8 24.5 32.5 37.7 39.0 39.6 26.9 33.7
0.057 39.8 40.7 41.5 34.4 38.1 36.3 37.2 37.9 28.3 33.5 34.4 35.8 36.9 24.5 30.6 35.8 37.2 37.9 26.9 32.1
0.072 38.5 39.5 40.5 34.3 36.9 34.9 35.8 36.6 28.3 32.4 33.0 34.4 35.5 24.4 29.4 34.4 35.9 36.7 26.9 31.1
0.088 37.4 38.4 39.4 34.0 35.8 33.9 34.8 35.7 28.2 31.5 31.8 33.2 34.3 24.4 28.4 33.3 34.8 35.6 26.8 30.1
0.103 36.6 37.7 38.7 34.0 34.9 33.0 34.0 34.8 28.2 30.8 31.0 32.4 33.4 24.3 27.6 32.3 33.8 34.6 26.8 29.6
0.119 35.8 37.0 37.9 33.8 34.2 32.3 33.4 34.1 28.1 30.2 30.1 31.3 32.5 24.4 26.8 31.5 33.1 33.9 26.8 29.0
0.134 35.0 36.4 37.1 33.4 33.5 31.6 32.7 33.3 28.1 29.7 29.4 30.7 31.6 24.3 26.2 30.8 32.4 33.1 26.7 28.6
0.150 34.4 36.1 36.2 33.3 33.0 31.2 32.4 32.5 28.0 29.3 28.9 30.2 30.6 24.3 25.7 30.3 31.9 32.2 26.7 28.3
Table 3: PSNR for denoising of four images (‘oranges’, ‘cars’, ‘windows’, ‘statue’) using several methods (B: BM3D; S: BM3D Stokes; P: PBM3D; Z: Zhao; F: Faisan) and several values of , the standard deviation of the noise, added. Bold indicates greatest PSNR. It can be seen that PBM3D is always the best performing method. The pattern continues with other images, the results (including those shown here) are plotted in figure 5.
Figure 5: PSNR for denoised images as a function of , the standard deviation of noise. Above each plot is the name of the image denoised, line colours correspond to different denoising algorithms. For the top row the PSNR values are shown in table 3. It can be seen that for all images and all values of PBM3D produces images with the greatest PSNR.

Figures 67 & 8 show the denoised images corresponding to the row of table 3, as well as the ground truth and noisy images. It can be seen that as well as providing the greatest PSNR value, the visual quality of the images denoised using PBM3D is the greatest of the methods tested. In all three figures the component for the images denoised using BM3D, BM3D Stokes and PBM3D appear very similar to the ground truth, with the image denoised using Faisan appearing to be slightly less sharp. The and components of the images denoised using BM3D and Faisan appear to have more denoising artefacts than those denoised using BM3D Stokes and PBM3D. In the components the images denoised using PBM3D have cleaner edges, which are more similar to the ground truth than components denoised using all of the other methods, this is highlighted in figure 11, which shows a close up of the ‘window’ images. The components denoised using PBM3D are notably more faithful to the original than the other methods, which can be seen in figures 11 & 10.

Figure 6: Polarization components of ‘oranges’ image after application of several denoising methods. G: ground truth; N: noisy; B: BM3D; S: BM3D Stokes; P: PBM3D; Z: Zhao; F: Faisan. Noise standard deviation, . Note that the images have been scaled such that black represents and white represents .
Figure 7: Polarization components of ‘cars’ image after application of several denoising methods. G: ground truth; N: noisy; B: BM3D; S: BM3D Stokes; P: PBM3D; Z: Zhao; F: Faisan. Noise standard deviation,
Figure 8: Polarization components of ‘window’ image after application of several denoising methods. G: ground truth; N: noisy; B: BM3D; S: BM3D Stokes; P: PBM3D; Z: Zhao; F: Faisan. Noise standard deviation,
Figure 9: ‘oranges’ image with noise of standard deviation (a high noise level), denoised using BM3D Stokes (S) and PBM3D (P) (G: ground truth). For both BM3D Stokes and PBM3D, the images are visually similar to the ground truth. For both methods however, the images are notably different, and the images are almost unrecognisable.
Figure 10: Close up of ‘cars’ image from figure 7 (G: ground truth; S: BM3D Stokes; P: PBM3D). components are similar for the images denoised using BM3D Stokes and PBM3D, with slight differences noticeable in the car’s bumper. The detail around the number plate of the car and panels on the right side of the image are more faithfully denoised using PBM3D than BM3D Stokes.
Figure 11: Close up of ‘windows’ image from figure 8 (G: ground truth; S: BM3D Stokes; P: PBM3D). The component of the image denoised using PBM3D exhibits fewer artefacts than the imaged denoised using BM3D Stokes, especially underneath the window. In the components, the lower windows are much more faithfully represented by the image denoised using PBM3D than BM3D Stokes.

5.4 Denoising real polarization imagery

To further test PBM3D with real, rather than simulated noise (as has been used so far), we used a DSLR camera with a rotatable polarizer to capture the three camera components, , , , of a scene of several lab objects, using several exposure settings on the camera (table 4). The exposure setting was varied in order to vary the amount of noise present. The polarization images were then denoised using PBM3D. Figure 12(a) shows the of the captured image when the exposure was 0.0222s and figure 12(b) shows the of the same image, denoised using PBM3D. The effect of denoising is evident, with the perceptible noise in the noisy image being greater than for the denoised image.

Figure 12: image of a collection of lab objects, taken with an exposure of 0.0222s. (a): The image without denoising. (b): The image denoised using PBM3D. The circles indicate where the true value was measured using a spectrometer. It can be seen that the noisy image tends to show much larger values. The values measured at each point are shown in figure 13

In addition to the imaging polarimetry, we also measured the of several regions of the scene using a spectrometer. The intensity count was averaged across the wavelength range corresponding to the camera sensitivity (400-700nm) at three different orientations of a rotatable polarizer, , and . These mean intensities, , , , were then used to calculate the using equation 7. The of the corresponding regions in the polarization images was also calculated using equation 7 with a weighting on each of the camera components to account for the separate channels, , which corresponds to the luminance, , of the colorspace. The absolute difference between the values from the spectrometry and from the imaging polarimetry with the noisy image and the same image denoised using PBM3D are shown in figure 13. The results were that the process of denoising extended the range of exposure time over which the imaging polarimetric values were the same as the spectrometry measurements. Table 4 demonstrates that at an exposure time of s, when , the values of the from the noisy image become significantly different () from those calculated using the spectrometry measurements. In contrast, when the images were denoised using PBM3D the exposure time could be bought down to 0.0056s () before the values became different (). Therefore, denoising using PBM3D increases the accuracy of the measurements by reducing the effect of noise on the measurement, allowing approximately 3.5 times as much noise to be tolerated.

Figure 13: The absolute value of the difference between the values measured using the spectrometer and the noisy images, , and spectrometer and the denoised images, . The noisy and denoised images had varying noise values, as given by table 4. The locations where the values were calculated are shown in figure 12. The black line indicates where , it can be seen that using the denoised images for DoP measurements results in much greater agreement with measurements taken using the noisy image as for most measurements . Table 4 uses this data to show that denoising significantly reduces the error due to noise.
noisy denoised
Wilcoxon
exposure estimated
0.1667 0.0021 154 0.8934 217 0.6575
0.1000 0.0034 83 0.2700 191 0.6134
0.0500 0.0055 74 0.0620 145 0.9589
0.0222 0.0103 43 0.0024 157 0.9261
0.0111 0.0199 18 0.0001 116 0.4154
0.0056 0.0363 16 0.0000 73 0.0402
0.0029 0.0678 7 0.0000 75 0.0022
Table 4: Estimated , the standard deviation of noise, and Wilcoxon test results for the data in figure 13. values were estimated using the method in [16]. The Wilcoxon test indicates that when the values calculated from the noisy image are significantly different to the values calculated from the spectrometer (bold indicates ). In contrast, the values calculated from the denoised images are significantly different when . Denoising therefore significantly reduces the effect of noise when .

6 Conclusion

Imaging polarimetry provides additional useful information from a natural scene compared to intensity-only imaging and it has been found to be useful in many diverse applications. Imaging polarimetry is particularly susceptible to image degradation due to noise. Our contribution is the introduction of a novel denoising algorithm, ‘PBM3D’, that when compared to state of the art polarization denoising algorithms, gives superior performance. When applied to a selection of noisy images, those denoised using PBM3D had a PSNR of 4.50dB greater on average than those denoised using the method of Faisan et al. [16], and 0.84dB greater than those denoised using BM3D Stokes. PBM3D relies on a transformation from camera components into intensity-polarization components. We have given two algorithms for computing the optimal transformation matrix, and given the optimal for our system and dataset. We have also shown that if imaging polarimetry is used to provide point measurements, that denoising using PBM3D allows approximately 3.5 times as much noise to be present than without denoising for the image to still have accurate measurements.

7 Funding Information

EPSRC CDT in Communications (EP/I028153/1); Air Force Office of Scientific Research (FA8655-12-2112).

The authors thank C. Heinrich and J. Zallat for the use of their denoising code.

References

  • [1] E. Hecht, Optics (Addison-Wesley, 2002).
  • [2] G. Horváth and D. Varju, Polarized Light in Animal Vision: Polarization Patterns in Nature (Springer Science & Business Media, 2004).
  • [3] G. Horváth, ed., Polarized Light and Polarization Vision in Animal Sciences (Springer, 2014). DOI: 10.1007/978-3-642-54718-8.
  • [4] R. Wehner and M. Müller, “The significance of direct sunlight and polarized skylight in the ant’s celestial system of navigation,” Proc. Natl. Acad. Sci. U.S.A. 103, 12575–12579 (2006).
  • [5] M. J. How, M. L. Porter, A. N. Radford, K. D. Feller, S. E. Temple, R. L. Caldwell, N. J. Marshall, T. W. Cronin, and N. W. Roberts, ‘‘Out of the blue: the evolution of horizontally polarized signals in Haptosquilla (Crustacea, Stomatopoda, Protosquillidae),” J. Exp. Biol. 217, 3425–3431 (2014).
  • [6] M. J. How, J. H. Christy, S. E. Temple, J. M. Hemmi, N. J. Marshall, and N. W. Roberts, “Target Detection Is Enhanced by Polarization Vision in a Fiddler Crab,” Curr. Biol. 25, 3069–3073 (2015).
  • [7] J. Taylor, P. Davis, and L. Wolff, “Underwater partial polarization signatures from the shallow water real-time imaging polarimeter (shrimp),” OCEANS’02 MTS/IEEE pp. 1526–1534 (2002).
  • [8] T. York, S. B. Powell, S. Gao, L. Kahan, T. Charanya, D. Saha, N. W. Roberts, T. W. Cronin, J. Marshall, S. Achilefu, S. P. Lake, B. Raman, and V. Gruev, “Bioinspired Polarization Imaging Sensors: From Circuits and Optics to Signal Processing Algorithms and Biomedical Applications,” Proc. IEEE 102, 1450–1469 (2014).
  • [9] F. Snik, J. Craven-Jones, M. Escuti, S. Fineschi, D. Harrington, A. De Martino, D. Mawet, J. Riedi, and J. S. Tyo, “An overview of polarimetric sensing techniques and technology with applications to different research fields,” Proc. SPIE 9099, 90990B (2014).
  • [10] W. de Jong, J. Schavemaker, and A. Schoolderman, “Polarized Light Camera; a Tool in the Counter-IED Toolbox,” in “Prediction and Detection of Improvised Explosive Devices (IED) (SET-117),” (RTO, 2007).
  • [11] S.-S. Lin, K. Yemelyanov, E. N. Pugh, and N. Engheta, “Polarization enhanced visual surveillance techniques,” in “Proceedings of IEEE International Conference on Networking, Sensing and Control,” (2004), pp. 216–221.
  • [12] D. Miyazaki and K. Ikeuchi, “Shape estimation of transparent objects by using inverse polarization ray tracing.” IEEE Trans. Pattern Anal. Mach. Intell. 29, 2018–29 (2007).
  • [13] A. E. R. Shabayek, O. Morel, and D. Fofi, “Bio-Inspired Polarization Vision Techniques for Robotics Applications,” Handbook of Research on Advancements in Robotics and Mechatronics pp. 81–117 (2015).
  • [14] K. H. Britten, T. D. Thatcher, and T. Caro, ‘‘Zebras and Biting Flies: Quantitative Analysis of Reflected Light from Zebra Coats in Their Natural Habitat,” PLOS ONE 11, e0154504 (2016).
  • [15] Y.-Q. Zhao, Q. Pan, and H.-C. Zhang, “New polarization imaging method based on spatially adaptive wavelet image fusion,” Opt. Eng. 45, 123202–123202–7 (2006).
  • [16] S. Faisan, C. Heinrich, F. Rousseau, A. Lallement, and J. Zallat, “Joint filtering estimation of Stokes vector images based on a nonlocal means approach,” J. Opt. Soc. Am. 29, 2028 (2012).
  • [17] A. Buades, B. Coll, and J. Morel, “A Review of Image Denoising Algorithms, with a New One,” Multiscale Model. Simul. 4, 490–530 (2005).
  • [18] E. Collett, Field Guide to Polarization (SPIE, 1000 20th Street, Bellingham, WA 98227-0010 USA, 2005).
  • [19] J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, ‘‘Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45 (2006).
  • [20] J. Zallat, A. S, and M. P. Stoll, “Optimal configurations for imaging polarimeters: impact of image noise and systematic errors,” J. Opt. A: Pure Appl. Opt. 8, 807 (2006).
  • [21] R.-Q. Xia, X. Wang, W.-Q. Jin, and J.-A. Liang, “Optimization of polarizer angles for measurements of the degree and angle of linear polarization for linear polarizer-based polarimeters,” Opt. Commun. 353, 109–116 (2015).
  • [22] S. Chang, B. Yu, and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising,” IEEE Trans. Image Process. 9, 1522–1531 (2000).
  • [23] G. Sfikas, C. Heinrich, J. Zallat, C. Nikou, and N. Galatsanos, “Recovery of polarimetric Stokes images by spatial mixture models,” J. Opt. Soc. Am. 28, 465 (2011).
  • [24] J. Valenzuela and J. Fessler, “Joint reconstruction of Stokes images from polarimetric measurements,” J. Opt. Soc. Am. A 26, 962–968 (2009).
  • [25] J. Zallat and C. Heinrich, “Polarimetric data reduction: a Bayesian approach,” Opt. Express 15, 83 (2007).
  • [26] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering,” IEEE Trans. Image Process. 16, 2080–2095 (2007).
  • [27] H. Sadreazami, M. O. Ahmad, and M. N. S. Swamy, “A study on image denoising in contourlet domain using the alpha-stable family of distributions,” Signal Processing 128, 459–473 (2016).
  • [28] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Color Image Denoising via Sparse 3d Collaborative Filtering with Grouping Constraint in Luminance-Chrominance Space,” in “2007 IEEE International Conference on Image Processing,” , vol. 1 (2007), vol. 1, pp. I – 313–I – 316.
  • [29] A. Danielyan, A. Foi, V. Katkovnik, and K. Egiazarian, “Denoising of Multispectral Images via Nonlocal Groupwise Spectrum-PCA,” Conference on Colour in Graphics, Imaging, and Vision 2010, 261–266 (2010).
  • [30] M. Maggioni, V. Katkovnik, K. Egiazarian, and A. Foi, “Nonlocal Transform-Domain Filter for Volumetric Data Denoising and Reconstruction,” IEEE Trans. Image Process. 22, 119–133 (2013).
  • [31] M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian, “Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal Transforms,” IEEE Trans. Image Process. 21, 3952–3966 (2012).
  • [32] M. Lebrun, “An Analysis and Implementation of the BM3d Image Denoising Method,” Image Processing On Line 2, 175–213 (2012).
  • [33] L. B. Wolff, “Polarization-based material classification from specular reflection,” IEEE Trans. Pattern Anal. Mach. Intell. 12, 1059–1071 (1990).
  • [34] Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “Polarization-based vision through haze.” Appl. Opt. 42, 511–25 (2003).
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
1586
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description