Deep Underwater Image Enhancement
In an underwater scene, wavelength-dependent light absorption and scattering degrade the visibility of images, causing low contrast and distorted color casts. To address this problem, we propose a convolutional neural network based image enhancement model, i.e., UWCNN, which is trained efficiently using a synthetic underwater image database. Unlike the existing works that require the parameters of underwater imaging model estimation or impose inflexible frameworks applicable only for specific scenes, our model directly reconstructs the clear latent underwater image by leveraging on an automatic end-to-end and data-driven training mechanism. Compliant with underwater imaging models and optical properties of underwater scenes, we first synthesize ten different marine image databases. Then, we separately train multiple UWCNN models for each underwater image formation type. Experimental results on real-world and synthetic underwater images demonstrate that the presented method generalizes well on different underwater scenes and outperforms the existing methods both qualitatively and quantitatively. Besides, we conduct an ablation study to demonstrate the effect of each component in our network.
Acquisition of clear underwater images is of great importance for ocean engineering and ocean research where autonomous and remotely operated underwater vehicles are widely used to explore and interact with marine environments. However, raw underwater images seldom meet the expectations concerning image visual quality. Naturally, underwater images are degraded by the adverse effects of light absorption and scattering due to particles in the water including micro phytoplankton colored dissolved organic matter and non-algal particles . When the light propagates in an underwater scenario, the light received by a camera is mainly composed of three types of light: direct light, forward scattering light, and backscattering light. The direct light suffers from attenuation resulting in information loss of underwater images. The forward scattering light has a negligible contribution to the blurring of the image features. The backscattering light reduces the contrast of underwater images and suppresses fine details and patterns. Additionally, the red light first disappears, followed by the green and blue lights (the wavelengths of the red, green and blue lights are 600nm, 525nm, and 475nm, respectively). As a result, most underwater images are dominated by a bluish or greenish tone. Figure 1 presents a diagram of light attenuation with respect to the wavelength of light.
These absorption and scattering problems hinder the performance of underwater scene understanding and computer vision applications such as aquatic robot inspection  and marine environmental surveillance . Therefore, it is necessary to develop effective solutions to improve the visibility, contrast, and color properties of underwater images for a superior visual quality and appeal.
Notable progress has been made to improve the visual quality of underwater images in recent years (see  for a survey). Existing underwater images sharpness methods can be classified into one of three broad categories: image enhancement methods, image restoration methods, and supplementary-information specific methods.
The image enhancement techniques modify the image pixel values to produce a subjectively and aesthetically pleasing image for specific objectives (e.g., contrast improvement, denoising, and brightness enhancement) without relying on any physical imaging models. In this line of research,  presented a fusion-based method to improve the visual quality of underwater images and videos. The method of  fuses a contrast improved underwater image and a color corrected underwater image obtained from input. In the process of mulit-scale fusion, four weights are used to determine which pixel is advantaged to appear in the final image. This method improves the global contrast and visibility; however, some regions in the resultant images become over-enhanced or under-enhanced. [8, 9] suggested a Rayleigh-stretched contrast-limited adaptive histogram method which is a modified version of ’s method. This method constrains the number of under-enhanced and over-enhanced regions. Nevertheless, it also tends to increase noise in the resultant images. A hybrid method based on color correction and underwater image dehazing for underwater image enhancement was proposed in , which corrects the color casts of underwater image using image color prior and improves the visibility by a modified image dehazing algorithm. This method shows limitations when the image color prior is not available.  proposed an underwater image color correction method based on weakly supervised color transfer, which learns a cross domain mapping function between underwater images and air images. This method relaxes the need for paired underwater images for training and allows the underwater images being taken in unknown locations. Recently,  modified their previous work  in order to reduce the effects of the over-enhancement and over-exposure.
The image restoration methods (e.g., , , , , , , , , , ) consider the challenge at hand as an inverse problem, and construct physical models of the degradation, and then estimate model parameters.  proposed a prior that exploited the difference in attenuation among the three color channels to predict the medium transmission of underwater scene. As a result, the effects of light scattering in underwater image are removed. However, such a method shows limitations when it is used to process underwater images which have not the strong difference in attenuation among three color channels of an underwater image.  combined an image dehazing algorithm with a wavelength dependent compensation algorithm to restore underwater image, which can remove the bluish tone of underwater images and the effects of the artificial light. However, this method is limited in processing the underwater images with serious color casts. A Red Channel method presented by  recovered the lost contrast of the underwater image by restoring the colors associated with short wavelengths. Similarly,  proposed an underwater dark-channel prior called UDCP which modifies the dark channel prior presented in . With the proposed UDCP, the medium transmission can be estimated in some cases; however, underwater dark-channel prior does not always hold when there are white objects or artificial light in the underwater scenes.  combined an underwater image dehazing algorithm with a contrast enhancement algorithm. This method can yield two enhanced underwater images at the same time. One image with relative genuine color and natural appearance is suitable for display while another image with high contrast and brightness can be used for unveiling more image details. Recently,  proposed a CNN based real-time underwater image color correction model based on synthetic underwater images generated in the weakly supervised learning manner. Nevertheless, this method is only effective for underwater images captured under specific scenes that have to be similar to those of its training data. It is, thus, impractical for real applications. Very recently,  proposed a generalized dark channel prior, which estimates the medium transmission by calculating the difference between the observed intensity and the ambient light. After that, the degraded images are recovered according to an image formation model.
The supplementary-information specific methods such as , , , and  usually take advantage of the additional information obtained from the multiple images captured by polarization filters, stereo images, rough depth of the scene, or specialized hardware devices.
Despite these recent efforts, existing approaches have still the following issues:
The perceived contrast in the latent image is often erratically distributed, remaining ineffectively low in under-enhanced regions and distractingly high in over-enhanced regions. Moreover, some methods introduce artifacts.
Most methods directly employ generic outdoor haze models to predict the underwater imaging model parameters, which is not well-suited for marine scenarios as the nature of imaging and lighting in such environments are different.
The specialized sensors and use of multiple images can be prohibitive, expensive and time-consuming, which reduce their applicability.
Although deep learning has achieved impressive performance on low-level vision tasks, there is no effective deep model for underwater image enhancement. The main reason is due to lacking of sufficiently labeled training data, which limits the development of deep learning based underwater image enhancement methods.
In contrast, we propose a new underwater image synthesis method, and then design to offer a robust and data-driven solution to solve these issues. We propose a deep convolutional neural network, which is shown to have superior robustness, accuracy, and flexibility for varying water types for marine imaging applications.
|(a) Input||(b) Enhanced|
Figure 2 shows an example of the enhanced image for a typical real-world underwater image. As visible in Figure 2(b), our method can accurately recover the underlying color distribution and unveil the suppressed details.
Contributions: Inspired by deep learning based approaches for low-level visual tasks (e.g., image dehazing by , image de-raining by , image depth estimation by ), we design an end-to-end solution for our complex and nonlinear model using a novel convolutional neural network architecture. Our model robustly restores the degraded underwater images and accurately reconstructs underlying colors and appearance. To summarize,
We introduce a novel convolutional neural network model to reconstruct the clear latent underwater image while preserving the original structure and texture by jointly optimizing MSE and SSIM losses. Unlike other mapping function objective minimizing CNN based approaches, our network learns the difference between the degraded underwater image and its clean counterpart.
We incorporate a new underwater image synthesis method that is capable of simulating a diverse set of degraded underwater images for data augmentation. To our best knowledge, it is the first underwater image synthesis method that can simulate different underwater types and degradation levels. Our image synthesis can be used as a guide for subsequent network training and full-reference image quality assessments, which calls for the development of new underwater image enhancement methods.
Our method is a fully data-driven and end-to-end model. It attains the state-of-the-art performance and generalizes well both on synthetic and real-world underwater images with varying color and visibility characteristics. Hence, our method is suitable for practical applications.
In this section, we provide the physical model of underwater image formation and the reason for using CNN as a prior
Ii-a Underwater Image Formation
Following the formulation in , the underwater image of light after scattering can be expressed as
where is the captured underwater image, is the clear latent image, also called as the scene radiance, that we aim to recover, is the homogeneous global background light, is the wavelength of the light for the red, green and blue channels, and is a point in the underwater scene (for clarity, images are denoted in bold capital letters). The medium energy ratio represents the percentage of the scene radiance reaching at the camera after reflecting from the point in the underwater scene, which thereby causes color cast and contrast degradation. In other words, is a function of the wavelength of the light and the distance from scene point to the camera
where is the wavelength-depended medium attenuation coefficient as shown in Figure 1. Assuming the energy of a light beam emanated from before and after it passes through a transmission medium at a distance of is and , respectively. The normalized residual energy ratio corresponds to the ratio of residual energy to initial energy for every unit of distance propagated. Its value varies in water depending on the light wavelength. For example, red light possesses a longer wavelength thus it attenuates faster and gets absorbed more than other wavelengths in water, which results in a bluish tone of most underwater images (see  for an extended discussion).
Ii-B Formulation as an Image Restoration Task
To recover the latent image , the conventional underwater image restoration methods attempt to estimate not only but also and from a single underwater image , which constitutes an underdetermined system as there are fewer equations than the unknowns. Conventional methods also proceed through two main steps. First, they estimate the model parameters of the homogeneous global background light and the medium energy ratio . After the model parameter estimation, they reconstruct the latent image by inverting the underwater formation model.
Unlike the previous methods, we directly estimate the latent image without calculating the homogeneous global background light and the medium energy ratio and we treat the underwater image enhancement as an image restoration task. Instead of direct estimation, our method recovers the residual between the target latent image and the given underwater image in a data-driven and end-to-end manner. Image restoration is an ill-posed problem due to the loss of information, which results in an underdetermined system. To allow a unique solution in the solution space, image restoration methods resort to imposing (often heuristic) regularization schemes or application and domain specific priors.
For a given the underwater image , we find the most likely reconstruction of the latent image using a maximum a posteriori (MAP) estimator assuming the relation between these two images in terms of a nonlinear function . This suggests a maximization over the probability distribution of the posterior using the Bayes rule
where is the likelihood of observing given that is the scene radiance and is the prior on the latent image. Having a uniform distribution on the observations, the maximization of the posterior
can be obtained by minimizing the negative log likelihood as
where the quadratic data fidelity term enforces the observations to be faithful to the degraded version of the estimated latent image, and is a trade-off parameter between the contributions of the prior and the data fidelity terms. The regularization prior guides the optimization process to the desired output. To model , existing image restoration methods use total variation (), Gaussian mixture model (), K-SVD ( and tailored class-specific priors ( ). Reformulating the underwater image enhancement as an image restoration problem with an explicit regularization prior offers several benefits; i) The exact parameterization of the prior can be unknown in solving Eq (5). ii) Alternative image priors can be exploited jointly. iii) Such priors can be plugged into other similar inverse problems.
Ii-C Why CNN?
Our choice for image restoration and regularization in Eq (5) is a cascade of hierarchical nonlinear filtering with progressively increasing local receptive fields, which are modeled by the convolutional neural networks without spatial pooling. Our model imposes a natural bottleneck to the residual between the latent and underwater images as its number of parameters is much smaller than the number of input pixels yet its patch filters allow guiding its enhancement factors based on the blur and color within local patches. Furthermore,
CNN provides a strong modeling capability of the distortions that we aim to compensate between the latent and original underwater images and facilitates discriminative prior learning.
It models the external priors by learning the relationship between the underwater image and the ground truth, while many existing methods attempt to model internal priors relying on the underwater image statistics and features e.g., ODM  and are complementary to our method. By combining our approach with ODM  is expected to improve the performance.
Inference on a CNN can be made efficiently by exploiting the parallel processing platforms.
Iii Our Proposed Model
Here, we discuss the details of the proposed underwater image enhancement fully Convolutional Neural Network (UWCNN) and then present a post-processing stage to further improve our enhanced result.
Iii-a Network Architecture
Figure 3 shows the architecture of our underwater image enhancement model UWCNN. It is a densely connected fully convolutional network that is inspired by the recent densely connected network models in object classification . In the following section, we present its basic building blocks and hyperparameters. The input to our network is a RGB underwater image .
Residuals: Unlike the conventional approaches that directly predict the clean latent image by learning the mapping function , we allow our network to learn the difference between the synthetic underwater image and its clean counterpart. Note that such a synthetic image generation task is a nontrivial objective for underwater image enhancement and restoration field, and it will be discussed in detail in Section IV-A. As underwater image and its feature maps in the subsequent layers are processed through many convolutional filters before reaching the final loss layer. Although our network is not intentionally very deep, there is still a possibility of vanishing or exploding gradients as mentioned in . To avoid such issues during the training iterations, we enforce learning the residue by adding the input of the network i.e. to the output of the network i.e. (see below) before loss function as
where “+” is the element-wise addition operation.
Blocks: The UWCNN has a modular architecture composed of blocks having same structure and components. Suppose and are the notation for ReLU and convolution, then the first operation of convolution and ReLU pair, in the -th block, is given by
where is the output of the first convolution-ReLU pairs of -th residual block and is a set of weights and biases associated with it. By composing the series of convolution-ReLU pairs, we obtain
The output of the -th block is obtained by concatenating along third dimension of each individual convolution-ReLU pairs output ’s and input image as
The output of the -th block is obtained by concatenating along the third dimension of each convolution-ReLU pairs output ’s and input image as
Finally, we chain all blocks and the output of this chain is convolved with a final convolution layer with parameters to predict the component as .
Network Layers: Our fully convolutional network consists of three different layers indicated by different colors as shown in Figure 3. The first type is of convolutional layers represented by “Conv” in Figure 3, which consists of 16 convolutional kernels of size to produce 16 output feature maps for the first layer, while subsequent convolutional layers produce 16 maps each using filters.
The second type of the layers is activation layers, also known as “ReLU”, to introduce the nonlinearity. The third type is “Concat” layers, which is used to concatenate all the convolutional layers after each block. The last convolutional layer estimates the final output of the network, which is the latent image.
Dense Concatenation: We stack all convolutional layers at the end of each block. This technique is different from DenseNet provided in , where each convolutional layer is connected with other convolutional layers in the same block. Furthermore, we do not use any fully connected layers or batch normalization steps, which makes our network memory efficient and fast. In addition, we feed the input image to every block as can be seen from connections of Figure 3. The stacking of the convolutional layers with input data reduces the need for a very deep network, and this property of stacking layers can be attributed to the superior performance of our proposed system. In summary, our network is unique since:
the input image is applied to all blocks, and
it contains only the fully-convolutional layers without any batch-normalization steps.
Network Depth: Our network is of modular structure and consists of three blocks where each block is again composed of three convolutional layers. We have a single convolutional layer at the end of the network, hence, making the full depth of our network only ten layers. This makes our model computationally inexpensive and highly practical in training as well as during prediction.
Reducing Boundary Artifacts: In end-to-end low-level vision tasks, the output of the system is needed to be equal to the input of the system. This requirement sometimes results in boundary artifacts. To avoid this phenomenon, we enforce two strategies:
we do not use any pooling layers in our network, and
we add zeros before each convolutional layer.
As a consequence, the final output image of UWCNN network is almost artifacts-free around the boundaries and is of the same size as the input image.
Iii-B Network Loss
Since we aim to reconstruct an image, we compute the objective function loss pixel-wise. We use the loss, i.e., MSE (Mean Square Error) as in our observations it can well preserve the sharpness of edges and details. Blurring edges results in large errors. We add the estimated residual to the input underwater image, then compute the MSE loss as
where is the estimated latent image pixel value at , as described in Eq. (6) and is the corresponding ground truth image in the training dataset.
In addition, we include the SSIM loss  in our objective function to impose the structure and texture similarity on the latent image. We use the gray images to compute SSIM scores. For each pixel , the SSIM value is computed within a image patch around the pixel as
where and correspond to the mean and standard deviation of the image patch from the latent image , similarly, and are for the patch from the ground truth image . The cross-covariance is computed between the patches from and for the pixel . We set constants and based on the default in SSIM loss. Our model is insensitive to these defaults. Still, we fix them for a fair comparison. Finally, the SSIM loss is expressed as
The final loss function is the aggregation of MSE and SSIM losses
At last, we learn the residual mappings between the degraded input and the latent result. Instead of the classical stochastic gradient descent (SGD) optimizer (see ), we use ADAM (see ) to update our UWCNN weights iteratively. ADAM combines the advantages of AdaGrad (see ) that works well with sparse gradients, and RMSProp (see ) that works well in on-line and non-stationary settings, which is relatively easy to configure where the default configuration parameters do well on most problems. Following the popular settings in low-level vision task networks, we assign the learning rate to 0.0002, ADAM’s internal parameters to 0.9 and to 0.999 (see  for their definition), and fix the learning rate in the entire training procedure.
UWCNN generates enhanced images without color casts and exceptional visibility. However, due to the limitation of our training data pairs (an indoors image as the latent image and a synthesized image from the indoors image using the aforementioned underwater image formation model as the corresponding underwater image), the enhanced images have lower dynamic range. In practice, one would expect the enhanced results to have vivid colors and higher contrast.
To solve this issue, we employ a simple yet effective adjustment as a post-processing stage. We denote UWCNN with post-processing as UWCNN+. The image is first transformed to HSI color space. Then, the ranges of its saturation and intensity components in the HSI color space are normalized to as
where and are the maximum and minimum saturation or intensity values in the UWCNN image. To avoid the effect of a single pixel having too high or too low values on the selection of maximum and minimum saturation or intensity values, we calculate the histogram distribution of saturation or intensity and select (frequency) as determinant. This means ignoring all the pixels values that frequency and only applying the normalization method to the frequency pixel values. is a heuristic value based on extensive experiments. After this simple saturation and intensity normalization, we transform the modified result back to RGB color space.
Sample results are shown in Figure 4. As visible, UWCNN removes effectively the dominant greenish color distortion in these real-world underwater images and improves significantly the contrast while preserving natural look and authenticity of the images. Compared to UWCNN, the saturation and intensity normalization in UWCNN+ further improves the contrast and brightness, unveiling more details. Our methods do not introduce extra colors as some other existing methods either.
Iv Experimental Evaluations
To evaluate our method, we perform qualitative and quantitative comparisons with the recent state-of-the-art underwater image enhancement methods on both synthetic and real-world underwater images. These methods include UDCP by , RED by , ODM by  and UIBLA by . We run the source codes provided by the corresponding authors with the recommended parameter settings to produce the best results for an objective evaluation. Since WaterGAN  is only applicable to specific scenarios, its results are not competitive and therefore not reported in this paper. For real-world images where the light-attenuation coefficients are not available, we apply each of the ten UWCNN models we learned and present the results that are visually more appealing. This process can be improved by using a classification stage to choose the best model, which we plan to explore as future work. For synthetic data, we present the results without post-processing since the models are derived from the synthetic data thus no intensity and saturation normalization is required. At last, we conduct an ablation study to demonstrate the effect of each component of our network.
Unlike the high-level visual analytics tasks where large training datasets are often available, the underwater image enhancement depends on the synthetic data. Next, we explain how we generate the synthetic underwater images. To the best of our knowledge, this is the first physical model based underwater image synthesis method that can simulate a diverse set of water types and degradation levels, which is a significant contribution for the development of data-driven underwater image enhancement techniques.
Iv-a Underwater Image Synthesis
|clear image||depth map||Type-1||Type-3||Type-5||Type-7|
To synthesize the training set for our UWCNN model, we use the attenuation coefficients described in  for the different water types of oceanic and coastal classes (i.e., I, IA, IB, II, and III for open ocean waters, and 1, 3, 5, 7, and 9 for coastal waters). As mentioned before, Type-I is the clearest and Type-III is the most turbid open ocean water. Similarly, for coastal waters, Type-1 is the clearest and Type-9 is the most turbid. We apply Eq (1) and Eq (2) to build ten types of underwater image datasets using the RGB-D NYU-v2 indoor dataset of , which consists of 1449 images. We select the first clean images for the training set and the remaining clean images for the validation (test) set.
To synthesize an underwater image, we generate a random homogeneous global atmospheric light . Then, we vary the depth from 0.5m to 15m, which is followed by the selection of the corresponding values of the red, green, and blue channels for the water types presented in Table I. For each image, we generate five underwater images based on random and depth ; therefore, we obtain a training set of 5000 and a validation set of 2495 samples. For computational efficiency, we resize these images to . In total, we synthesize ten underwater image datasets and train ten UWCNN models for each type.
Figure 5 shows these ten different types of underwater images for a sample. It is evident that the underwater images of Type-I, Type-IA and Type-IB are similar in their physical appearance and characteristics. Thus, we select a total of eight models out of ten to display the results on synthetic underwater images.
Iv-B Network Implementation and Training
For training, the input to our network is the synthetic images of size generated from the NYU-v2 RGB-D dataset without any augmentation or preprocessing. We trained our model using ADAM (see ) and set the learning rate to 0.0002, to 0.9, to 0.999. The batch size is set to 16. It takes around three hours to optimize a model over 20 epochs. We use TensorFlow as the deep learning framework on an Inter(R) i7-6700k CPU, 32GB RAM, and a Nvidia GTX 1080 Ti GPU.
|(a) Real images||(b) RED||(c) UDCP||(d) ODM||(e) UIBLA||(f) UWCNN||(g) UWCNN+|
Iv-C Synthetic Underwater Images
We first evaluate the capacity of our method in terms of quantitative metrics, thus we exploit synthetically generated underwater images, which is common in other computer vision tasks as they show the capacity of an algorithm in terms of quantitative metrics. We compare the UWCNN model with several state-of-the-art methods. In Figure 6, we present the results of underwater image enhancement on the synthetic underwater images from our validation set.
As shown in Figure 6(a), the synthetic underwater images accord with the measurement of . The RED method of  is effective for the clear types i.e., Type-1, Type-3, Type-5, and Type-I; however, for turbid one’s i.e., Type-7, Type-9, Type-II, and Type-III, it leaves haze artifacts in those images, moreover, it introduces color deviations. Similarly,  produces distinctly darkish results while ODM by  and UIBLA of  improve the visibility in their respective outcomes while introducing artificial colors or color deviations. On the other hand, our UWCNN not only enhances the visibility of the images but also restores an aesthetically pleasing texture and vibrant yet genuine colors. In comparison to other methods, the visual quality of the UWCNN results resemble the ground-truth labels as shown in Figure 6(g).
Furthermore, we quantify the accuracy of the recovered images on the synthetic validation set including 2495 samples for each type. In Table II, the accuracy is measured by three different metrics: mean square error (MSE), peak signal to noise ratio (PSNR), and the structural similarity index metric (SSIM) . In case of MSE and PSNR metrics, the lower MSE (higher PSNR) denotes the result is more close to the label in terms of image content. In case of the SSIM metric, the higher SSIM scores mean the result is more similar to the label in terms of image structure and texture. Here, the presented results are the average scores. The values in bold represent the best results.
As visible, among all underwater image enhancement methods we tested, UWCNN model comes out as the best performer across all metrics and all degradation types, demonstrating its effectiveness and robustness. Regarding the SSIM response, our method is at least better than the second-best performer. Similarly, our PSNR is higher (less erroneous as indicated by the MSE scores) than the compared methods. The results in Table 2 also agree with the subjective images of Figure 6.
Iv-D Real-world Underwater Images
We also evaluate the proposed method on real-world underwater images. Visual comparisons with competitive methods are presented in Figure 7. The real-world underwater images are gathered from the Internet because there is no public underwater image dataset available. These real-world underwater images have varying tone, light, and contrast.
A first glance at Figure 7 may give the impression that the results of ODM and UIBLA might be sharper; however, a careful inspection reveals that the ODM method causes over-enhancement and over-saturation (besides color casts) because the histogram distribution prior used in the ODM is not always valid. Similarly, the images produced by the UIBLA are not natural and consist of over-enhancement, a shortcoming of this method as the robustness value of the background light and the medium transmission score estimated by the prior are suboptimum. Figure 8 and Figure 9 show the failure cases of the ODM and UIBLE methods. The methods of RED and UDCP have little effect on the inputs. In contrast, our UWCNN+ shows promising results on real-world images, without introducing any artificial colors, color casts, over- or under-enhanced areas. We do agree that our method does not enhance real-world images as accurately as it does the synthetic ones, which can be improved by having more underwater images in model reconstruction.
Observing the failure cases in Figure 8 and Figure 9, one can find that the ODM method tends to introduce extra colors (e.g., the reddish color around the coral in Figure 8) while our approach improves the contrast, similar performance to the ODM, but maintains a genuine color distribution of the original underwater image. For the failure case of the UIBLA method in Figure 9, it aggravates the greenish color and produces visually unpleasing results. In contrast, our method removes the color casts and improves the contrast and brightness, which generates better visibility and a pleasant perception.
We note that the assessments in [46, 47] are slanted toward over-exposure or over-enhancement, where the histogram equalization method is regarded to yield better scores. For a more objective assessment, in addition to evaluating underwater image quality with several metrics, we conduct a user study to provide realistic feedback and quantify the subjective visual quality. We collect 20 real-world underwater images from the Internet and related papers. We show samples from this dataset in Figure 10. Some corresponding results are presented in Figure 7.
For the user study, we apply all methods to generate enhanced underwater images, randomize the order the results, and then display them on a screen to human subjects. There were 20 participants with image processing and computer vision expertise. Each subject ranks the results based on the perceived visual quality from 1 to 5 where 1 is the worst and 5 is the best. One expects that the results with high contrast, good visibility, natural color and authentic texture should receive higher ranks while the results with over-enhancement/exposure, under-enhancement/exposure, color casts, and artifacts should have lower ranks. The average subjective scores are given in Table III. Our UWCNN+ receives the highest rankings, which indicates that our method can produce better performance on real-world underwater images from a subjective visual perspective.
Iv-E Ablation Study
To demonstrate the effect of each component in our network, we carry out an ablation study involving the following experiments:
UWCNN without residual learning (UWCNN-woRL),
UWCNN without dense concatenation (UWCNN-woDC),
UWCNN without SSIM loss (UWCNN-woSSIM).
The quantitative evaluation is only performed on Type-1 and Type-III synthetic underwater image test sets due to the limited space. The average results in terms of MSE, PSNR, and SSIM are reported in Table LABEL:table:ablation. For MSE, a lower score is better. For PSNR and SSIM, the higher scores are better.
From Table LABEL:table:ablation, we see that replacing conventional learning strategy (UWCNN-woRL) with residual learning (UWCNN) could boost the performance. Comparing UWCNN with UWCNN-woDC, we observe that the dense concatenation also could improve the performance.
The use of SSIM loss (UWCNN) improves the structure and texture similarity at the cost of the decreased performance of MSE and PSNR (UWCNN-woSSIM). However, such a sacrifice is necessary for better subjective perception. Such an example is presented in Figure 11, which demonstrates the importance of SSIM loss. In Figure 11, after adding SSIM loss, the result of UWCNN has a more smooth background than that of UWCNN-woSSIM.
We presented convolutional neural network based underwater image enhancement network. We employ residual learning technique to train our models. Experiments are performed on synthetic and real-world images, which indicates the robust and effective performance of our method. To our advantage, our system only contains ten convolutional layers and 16 feature maps at each convolutional layer, which provides fast and efficient training and testing on GPU platforms.
To demonstrate the effectiveness of each component in our UWCNN network, we have carried out an ablation study. The results demonstrate that the residual learning, dense concatenation and SSIM loss used in our network boost the performance quantitatively and qualitatively.
In future, we plan to extend our current model to enhance hazy images and jointly produce image depth from a single hazy image. We will investigate using only one single model to predict the correct output from one single blind model of UWCNN to attain further accelerating in the process of UWCNN model enhancement.
Saeed Anwar received Bachelor degree in Computer Systems Engineering with distinction from University of Engineering and Technology (UET), Pakistan, in July 2008, and Master degree in Erasmus Mundus Vision and Robotics (Vibot), jointly from Heriot watt University United Kingdom (HW), University of Girona Spain (UD) and University of Burgundy France in August 2010 with distinction. During his masters, he carried out his thesis at Toshiba Medical Visualization Systems Europe (TMVSE), Scotland. He has also been a visiting research fellow at Pal Robotics, Barcelona in 2011. Since 2014, he is a PhD student at the Australian National University (ANU) and Data61/CSIRO. He has also been working as a Lecturer and Assistant Professor at the National University of Computer and Emerging Sciences (NUCES), Pakistan. His major research interests are low-level vision, image enhancement, image restoration, computer vision, and optimization.
Chong-yi Li received his Ph.D. degree with the School of Electrical and Information Engineering, Tianjin University, Tianjin, China in June 2018. From 2016 to 2017, he took one year study at the Research School of Engineering, Australian National University (ANU) as a visiting Ph.D. student supported by the CSC. Now, he is a lecture in the College of Electronic Information and Automation, Civil Aviation University of China (CAUC). His current research focuses on image processing, computer vision, and deep learning, particularly in the domains of image restoration and enhancement, such as images captured under the bad weather (hazy, foggy, sandy, dusty, rainy, snowy day) and special circumstances (underwater, weak illumination). He also focuses on other low-level vision problems, such as image/depth super-resolution reconstruction, image deblurring, image denoising, and multi-exposure image fusion.
Fatih Porikli is an IEEE Fellow and a Professor in the Research School of Engineering, Australian National University (ANU). He is acting as a Chief Scientist at Huawei. Previously, he led the Computer Vision Research Group at NICTA, and served as a Distinguished Research Scientist at Mitsubishi Electric Research Laboratories. He has received his PhD from New York University in 2002. Prof. Porikli is the recipient of the R&D 100 Scientist of the Year Award in 2006. He won 5 best paper awards at premier IEEE conferences and received 5 other professional prizes. He authored more than 200 publications and invented 73 patents. He is the co-editor of 2 books. He is serving as the Associate Editor of several journals. His research interests include computer vision, deep learning, manifold learning, online learning, and image enhancement with commercial applications in autonomous vehicles, video surveillance, consumer electronics, industrial automation, satellite and medical systems.
- C. Mobley, “Light and water: radiative transfer in natural waters,” Academic Press, 1994.
- D. Berman, T. Treibitz, and S. Avidan, “Diving into haze-lines: color restoration of underwater images,” in Proc. of British Machine Vision Conference (BMVC), pp. 1-12, 2017.
- N. Jerlov, “Marine optics,” 1976.
- G. Forest, “Visual inspection of sea bottom structures by an autonomous underwater vehicle,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 31, no. 5, pp. 691-705, 2001.
- N. Strachan, “Recognition of fish species by colour and shape,” Image and Vision Computing, vol 11, no. 1, pp. 2-10, 1993.
- R. Schettini and S. Corchs, “Underwater image processing: state of the art of restoration and image enhancement methods,” EURASIP Journal on Advances in Signal Processing, vol. 2010, pp. 1-14, 2010.
- C. Ancuti and C. O. Ancuti, “Enhancing underwater images and videos by fusion,” in Proc. of IEEE Int. Conf. Comput. Vis. Pattern Rec. (CVPR), 2012, pp. 81-88.
- A. Ghani and A. IsaN, “Underwater image quality enhancement through integrated color model with rayleigh distribution,” Applied Soft Computing, vol. 27, pp. 219-230, 2015.
- A. Ghani and A. IsaN, “Enhancement of low quality underwater image through integrated global and local contrast correction,” Applied Soft Computing, vol. 37, pp. 332-344, 2015.
- K. Iqbal, M. Odetayo, and A. James, “Enhancing the low quality images using unsupervised colour correction method,” in Proc. of IEEE International Conference on Systems, Man and Cybernetics, pp. 1703-1709, 2010.
- C. Li, J. Guo, C. Guo, R. Cong, and J. Gong, “A hybrid method for underwater image correction,” Pattern Recognition Letters, vol. 94, pp. 62-67, 2017.
- C. Li, J. Guo, and C. Guo, “Emerging from water: underwater image color correction based on weakly supervised color transfer,” IEEE Signal Processing Letters, vol. 25, no. 3, pp. 323-327, 2018.
- C. Ancuti and C. O. Ancuti and C. Vleeschouwer, “Color balance and fusion for underwater image enhancement,” IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 379-393, 2018.
- N. Carlevaris-Bianco, A. Mohan, and R. Eustice, “Initial results in underwater single image dehazing,” in Proc. of IEEE OCEANS, pp. 1-8, 2010.
- J. Chiang and Y. Chen, “Underwater image enhancement by wavelength compensation and dehazing,” IEEE Transactions on Image Processing, vol. 21, no. 4, pp. 1756-1769, 2012.
- A. Galdran, D. Pardo, A. Picn, and A. Alvarez-Gila, “Automatic red-channel underwater image restoration,” Journal of Visual Communication and Image Representation, vol. 26, pp. 132-145, 2015.
- C. Li and J. Guo, “Underwater image enhancement by dehazing and color correction,” Journal of Electronic Imaging, vol. 24, no. 3, pp. 033023-1-033023-10, 2015.
- P. Drews, E. Nascimento, and S. Botelho and M. Campos, “Underwater depth estimation and image restoration based on single images,” IEEE Computer Graphics and Applications, vol. 36, no. 2, pp. 24-35, 2016.
- C. Li, J. Guo, S. Chen, Y. Tang, Y. Pang, and J. Wang, “Underwater image restoration based on minimum information loss principle and optical properties of underwater imaging,” in Proc. of IEEE International Conference on Image Processing (ICIP), pp. 1993-1997, 2016.
- C. Li, C. Ji, R. Cong, Y. Pang, and B. Wang, “Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior,” IEEE Transactions on Image Processing, vol. 25, no. 12, pp. 5664-5677, 2016.
- Y. Peng and P. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE Transactions on Image Processing, vol. 26, no. 4, pp. 1579-1594, 2017.
- J. Li, K. Skinner, R. Eustice, and M. Roberson, “WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images,” IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 387-394, 2017.
- Y. Peng, K. Cao, and P. Cosman, “Generalization of the dark channel prior for single image restoration,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2856-2868, 2018.
- K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2341-2343, 2011.
- Y. Schechner and N. Karpel, “Clear underwater vision,” in Proc. of IEEE Int. Conf. Comput. Vis. Pattern Rec. (CVPR), 2004, pp. 536-543.
- Y. Schechner and N. Karpel, “Recovery of underwater visibility and structure by polarization analysis,” IEEE Journal of Oceanic Engineering, vol. 30, no. 3, pp. 570-587, 2005.
- T. Treibitz and Y. Schechner, “Active polarization descattering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 3, pp. 385-399, 2009.
- M. Sheinin and Y. Schechner, “The next best underwater view,” in Proc. of IEEE Int. Conf. Comput. Vis. Pattern Rec. (CVPR), 2015, pp. 3764-3773.
- B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “DehazeNet: an end-to-end system for single image haze removal,” IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187-5198, 2016.
- X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in Proc. of IEEE Int. Conf. Comput. Vis. Pattern Rec. (CVPR), 2017, pp.1715-1723.
- S. Anwar, Z. Hayder. and F. Porikli, “Depth Estimation and Blur Removal from a Single Out-of-focus Image,” in Proc. of British Machine Vision Conference (BMVC), pp. 1-12, 2017.
- A. Chambolle, “An algorithm for total variation minimization and applications,” Journal of Mathematical imaging and vision, vol. 20, no. 1-2, pp. 89-97, 2004.
- D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in Proc. of IEEE Int. Conf. Comput. Vis. (ICCV), 2011, pp. 479-486.
- M. Elad and M. Aharon “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736-3745, 2006.
- S. Anwar, C. Huynh, and F. Porikli, “Class-specific image deblurring,” in Proc. of IEEE Int. Conf. Comput. Vis. (ICCV), 2015, pp. 495-503.
- S. Anwar, F. Porikli, and C. Huynh, “Category-specific object image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 11, pp. 5506-5518, 2017.
- G. Huang, Z. Liu, L. van der Matten, and K. Weinberger, “Densely connected convolutional networks,” in Proc. of IEEE Int. Conf. Comput. Vis. Pattern Rec. (CVPR), 2017, pp. 2261-2269.
- Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157-166, 1994.
- H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Transactions on computational imaging, vol. 3, no. 1, pp. 47-57, 2017.
- Y. LeCun, L. Bottou, G. Orr, and K. Muller, “Efficient backprop,” Neural networks: Tricks of the trade, 1998.
- D. Kingma and J. Ba, “Adam: a method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2017.
- J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” The Journal of Machine Learning Research, vol. 12, pp. 2121-2159, 2011.
- T. Tieleman and G. Hinton, “COURSERA: Neural Networks for Machine Learning,” University of Toronto, 2012.
- N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in Proc. of Eur. Conf. Comput. Vis. (ECCV), pp. 746-760, 2016.
- Z. Wang, A. Bovik, H. Sherikh, and E. Simoncelli, “Image quality assessment: from error visibility to structual similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.
- M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 6062-6071, 2015.
- K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” IEEE Journal of Oceanic Engineering, vol. 41, no. 3, pp. 541-551, 2016.