FAMED-Net: A Fast and Accurate Multi-scale End-to-end Dehazing Network
Single image dehazing is a critical image pre-processing step for subsequent high-level computer vision tasks. However, it remains challenging due to its ill-posed nature. Existing dehazing models tend to suffer from model overcomplexity and computational inefficiency or have limited representation capacity. To tackle these challenges, here we propose a fast and accurate multi-scale end-to-end dehazing network called FAMED-Net, which comprises encoders at three scales and a fusion module to efficiently and directly learn the haze-free image. Each encoder consists of cascaded and densely connected point-wise convolutional layers and pooling layers. Since no larger convolutional kernels are used and features are reused layer-by-layer, FAMED-Net is lightweight and computationally efficient. Thorough empirical studies on public synthetic datasets (including RESIDE) and real-world hazy images demonstrate the superiority of FAMED-Net over other representative state-of-the-art models with respect to model complexity, computational efficiency, restoration accuracy, and cross-set generalization. The code will be made publicly available.
Images captured in hazy conditions often suffer from absorption and scattering effects caused by floating atmospheric particles such as dust, mist, and fumes, which can result in low contrast, blurry, and noisy images. This degraded image quality potentially challenges many subsequent high-level computer vision tasks, , object detection [1, 2, 3] and segmentation [4, 5, 6]. Therefore, removing haze and improving image quality benefits these applications, making image dehazing a subject of intense research and practical focus.
To be specific, image haze removal or dehazing refers to a technique that restores a haze-free image from a single or several observed hazy images. Many dehazing approaches have been proposed, which can be categorized into those that: 1) use auxiliary information such as scene depth  and polarization ; 2) use a sequence of captured images ; 3) use a single hazy image [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25], as the model input when dehazing. Of these, single image dehazing without the need for additional information is of most practical benefit. However, as a typical ill-posed problem, single image dehazing remains challenging and requires refinement.
The presence of haze leads to the combination of an attenuation term corresponding to the absorbing effect and a scattering term corresponding to the scattering effect that occur during imaging. Both terms are related to an intermediate variable, that is, transmission, which depends on scene depth. One feasible haze removal solution is to estimate the transmission and then recover the clear image by reversing the attenuation and scattering. Many single image dehazing methods have been proposed [13, 26, 14, 15, 16, 17, 18, 21], which use either hand-crafted features (, different image priors) or learning-based features to estimate the haze transmission.
For example, He et al.  proposed a simple and effective dark channel prior for single image dehazing, which assumes that the minimum of all the spectral channels in clear images (the “dark channel”) is close to zero. The method effectively estimates the haze transmission. However, the dark channel prior may not work for some particular scenes such as for white objects, which are similar to atmospheric light, because it underestimates the transmission and leads to over-dehazed artifacts. Zhu et al.  proposed a color attenuation prior that assumes a positive correlation between the scene depth and the haze concentration, which is represented by the subtraction of scene brightness from saturation. Then, the scene depth and haze transmission are easily estimated by a regressed linear model based on the above prior. Recently, Berman et al.  proposed a non-local prior based on the assumption that colors in a clear image can be approximated by some distinct colors clustering tightly in RGB space. Being affected by haze, each cluster becomes a line in RGB space (haze-line) due to the varying transmission coefficients of the clustered pixels. Consequently, the transmission and clear image are estimated according to these haze lines. Though prior-based methods are usually simple and effective for many scenes, they share the common limitation of describing specific statistics, which may not work for some images.
Learning-based methods adopt a data-driven approach to learn a linear/non-linear mapping between features and transmission and so overcomes the limitations of specific priors. For example, Tang et al.  proposed learning a regression model based on random forests from haze-relevant features including the dark channel, local max contrast, hue disparity, and local max saturation. They trained the model using a synthetic dataset and tested it on both synthetic and real-world hazy images, which then became common practice in subsequent learning-based methods. The learning-based idea for dehazing has subsequently been extended in three ways: 1) more powerful learning models; 2) more effective synthetic methods and larger datasets; 3) end-to-end modeling/training.
Deep neural networks have now been successfully applied to many computer vision tasks including object recognition, detection, and semantic segmentation. By leveraging their powerful representation capacity and end-to-end learning, many deep convolutional neural network (CNN)-based approaches were proposed for image dehazing [18, 19, 20, 21, 22, 23, 24]. For example, Cai et al.  proposed an end-to-end trainable deep CNN model called DehazeNet to directly learn the transmission from hazy images, which is superior to contemporary prior-based methods and random forest models . Ren et al.  proposed a multi-scale CNN (MSCNN) to learn the transmission map in a fully convolutional manner and explore a multi-scale architecture for coarse-to-fine regression.
Despite the effectiveness of CNN-based approaches, a separate step is still needed to estimate the atmospheric light. Recently, Zhang et al.  proposed an end-to-end densely connected pyramid dehazing network (DCPDN) to jointly learn the transmission map, atmospheric light, and dehazing. They adopted an encoder-decoder architecture with a multi-level pyramid pooling module to learn multi-scale features. They also utilized an adversarial loss based on a generative adversarial network  to supervise the dehazing network. Rather than estimating the intermediate transmission, Li et al . proposed an end-to-end CNN model called the all-in-one dehazing network (AOD-Net) to learn the clear image from a hazy one. They integrated the transmission and atmospheric light into a single variable by reformulating the hazy imaging model. Ren et al.  proposed a gated fusion network (GFN) by adopting an encoder-decoder architecture, while Li et al. also designed an encoder-decoder architecture but based on a conditional generative adversarial network (cGAN) to learn the dehazed image end-to-end. Though cGAN and DCPDN have achieved good dehazing results, they contain dozens of convolutional layers and are about 200 MB in size, making them awkward and unlikely to be applicable in the resource-constrained context of a computer vision system.
In this paper, we aim to develop a fast and accurate deep CNN model for single image dehazing. We use a fully convolutional and end-to-end training/testing approach to efficiently process hazy images of arbitrary size. To this end, we propose a fast and accurate multi-scale dehazing network called FAMED-Net, which comprises encoders at three scales and a fusion module to directly learn the haze-free image. Each encoder consists of cascaded point-wise convolutional layers and pooling layers via a densely connected mechanism. Since no larger convolutional kernels are used and features are reused layer-by-layer, FAMED-Net is lightweight and computationally efficient. Thorough empirical studies on public synthetic datasets and real-world hazy images demonstrate the superiority of FAMED-Net over representative state-of-the-art models with respect to model complexity, computational efficiency, restoration accuracy, and cross-set generalization. The code will be made publicly available at https://github.com/chaimi2013/FAMED-Net.
The main contributions of this paper can be summarized as follows:
We devise a novel multi-scale end-to-end dehazing network called FAMED-Net, which implicitly learns efficient statistical image priors for fast and accurate haze removal from a single image.
FAMED-Net leverages fully point-wise convolutions as the basic unit to construct the encoder-decoder architecture, which has a small model size and is computationally efficient.
FAMED-Net outperforms state-of-the-art models on both synthetic benchmarks and real-world hazy images.
Ii Related Work
Ii-a Atmospheric Scattering Model
where is the observed hazy image, is the scene radiance, is the atmospheric light assumed to be a global constant, is the haze transmission, denotes pixel location, and denotes the spectral channel, , . The first term, called the attenuation term, represents the haze absorbing effect on scene radiance, while the second term, called the scattering term, represents the haze scattering effect on ambient light. describes the fraction of scene radiance reaching the camera sensor, so is the “transmission”, which depends on scene depth. Under the homogeneous haze assumption, the transmission can be expressed as:
where denotes the medium attenuation coefficient and is the scene depth.
Ii-B Prior-based and Learning-based Image Dehazing Methods
As can be seen from the atmospheric scattering model in Eq. (1), given an observed hazy image , recovering the scene radiance is ill-posed. Different image priors have been proposed to constrain the haze-free image and make the estimate tractable, including the dark channel prior , color attenuation prior , and non-local prior , etc. As defined in , each pixel value of the dark channel refers to the minimum pixel value on each patch centered at every pixel position. Figure 1 shows an example of the dark channels on both clear and hazy images. As can be seen, the dark channel of a clear image is almost dark everywhere except for the bright sky region, while the dark channel of a hazy image reveals the haze veil due to the haze scattering effect (corresponds to the second term in Eq. (1)). Based on the dark channel prior, the transmission can be efficiently estimated from the dark channel map. It is noteworthy that the pixel value of the dark channel reveals the hazy density (which is related to scene depth) even though it is calculated locally in a sliding window manner (See the regions and corresponding values indicated by the red boxes). It can be explained as follows: 1) the haze effects of both attenuation and scattering which are directly related to scene depth, can be described as a pixel-to-pixel (, locally) mapping from clear pixel to hazy pixel by the atmospheric scattering model. 2) the dark channel prior reveals the intrinsic locally statistical property of clear images. Similar to , our approach also solves the dehazing problem in a local manner which implicitly learns a statistical image prior as will be demonstrated in Section IV-C4.
To overcome the limitations of prior-based methods, many deep CNN-based data-driven dehazing models have been proposed since Cai et al.  proposed DehazeNet, including MSCNN , AOD-Net , FPCNet , DCPDN , GFN , cGAN  and proximal DehazeNet . These can be categorized into those that: 1) estimate using CNN [18, 19, 21, 23, 30]; 2) directly learn the scene radiance end-to-end [20, 23, 22, 24, 30]. Our proposed method falls into the latter category and is partly inspired by AOD-Net  and FPCNet . In contrast to AOD-Net, we propose a fully point-wise CNN to regress and produce a stronger representation capacity. In contrast to FPCNet, we propose: 1) an end-to-end model to regress the scene radiance directly; 2) a multi-scale architecture to handle the scale variance, which achieves much better results than FPCNet while maintaining low model complexity and high computational efficiency; and 3) a new training/testing strategy that negates the need for a pre-processing shuffling step. Compared to MSCNN, in which coarse-scale predictions are used as part of the input for the finer scale, the proposed method adopts a Gaussian pyramid architecture and follows a late fusion strategy. It produces better dehazing results than MSCNN and runs faster. Compared to the recently proposed DCPDN and cGAN, our model is much more compact, , less than 90 kb vs. about 200 Mb, while having a high restoration accuracy and computational efficiency.
Ii-C Multi-scale pyramid architecture
The pyramid structure is a basic idea used for both multi-resolution image representation and multi-scale feature representation in the computer vision area, for example, Gaussian pyramid, Laplacian pyramid, wavelet , and SIFT . Leveraging this classical idea, CNN produces a feature pyramid through stacked convolutional layers and spatial pooling layers. Recently, different multi-scale image or feature pyramid architectures have been devised for both low- and high-level computer vision applications, including deep Laplacian pyramid networks for image super-resolution , DeepExposure using Laplacian pyramid decomposition , deep generative image models , Laplacian pyramid reconstructive adversarial network , Deeplab using an image pyramid for semantic segmentation , and feature pyramid networks for object detection . Our approach also adopts the Gaussian/Laplacian pyramid architectures for multi-scale fusion (See Figure (a)a and Figure (b)b). In contrast to those above methods, the proposed FAMED-Net is specifically devised for single image dehazing. Moreover, it leverages fully point-wise convolutions instead of convolutions with large kernels for constructing a lightweight and computationally efficient network.
Ii-D Deep Supervision
Adding auxiliary supervision on intermediate layers within a deep neural network also known as deep supervision is originally proposed by Xie and Tu in the seminal work [39, 40]. This technique facilitates multi-scale and multi-level feature learning by allowing error information backpropagation from multiple paths and alleviating the problem of vanishing gradients in deep neural networks. Deep supervision has been widely adopted in the following work in different areas such as Deeplab for semantic segmentation , MSCNN for image dehazing , LapSRN for image super-resolution , etc. We also add supervision on the dehazed image at each scale by leveraging the deep supervision idea.
Iii FAMED-Net for Single Image Dehazing
Iii-a A Probabilistic View to Solving the Ill-posed Dehazing Problem
Applying a logarithmic operation to both sides of the above equation produces the following general form:
where is the observed degraded image, is the ground truth haze-free image, and is the intermediate variable related to the degrading process. and can be estimated using maximum a posteriori estimation (MAP), ,
is the data likelihood, which corresponds to the data fidelity term measuring the reconstruction error. When using the L2 loss to supervise network training, it indeed assumes a normal distribution about the reconstruction error (see Section III-B and the yellow circle in Figure 2). The L1 loss can also be used to enforce a sparse constraint. is the conditional distribution of conditioned on the clear haze-free image. For example, DCP  assumes that (, ) concentrates on zeros. As with DehazeNet  and AOD-Net , the networks can implicitly learn and , as we will show in Section IV-C4. is the prior distribution of , usually assumed to be long-tailed due to the spatial continuity in natural images (locally smooth regions and sparse abrupt edges) . Markov Random Fields or simple filters like guided filter are used to model the spatial continuity .
Based on the above analysis, the key is to construct a model that can effectively learn statistical regularities. As shown in , statistical regularities in natural images can be efficiently learned by point-wise convolutions, which are compact and resists over-fitting. Partly inspired by , we devise a novel end-to-end fully point-wise CNN for single image dehazing.
Iii-B The Single-scale FAMED-Net: FAMED-Net-SS
As shown in Figure 2, the network is designed to learn the reformulated variable in Eq. (3) and recover the scene radiance according to Eq. (4) (see ). There are five point-wise convolutional layers, in which the first four form the K-encoder and the last forms the decoder. Features corresponding to different receptive fields are reused via dense connections (see black arcs and cubes in Figure 2). Mathematically, this can be formulated as:
where represents the learned features from the block. We denote the input as the block, the hazy image of size as , and the decoded features in the block as , , . denotes the index set, which indexes the feature maps used by the block via dense connections (), , , , , , in the proposed network. denotes the mapping function in the block learned by a combination of a convolutional layer, a batch normalization layer, a ReLU layer and a pooling layer.
We leverage pooling layers of different kernel sizes () after each convolutional layer to aggregate multi-level statistics (features) within the receptive fields, , . It is noteworthy that by using a combination of point-wise convolutional layers and a pooling layer, the output node has a receptive field of , which is equivalent to the one using a convolutional layer alone. In this way, we retain the representation capacity of the neural network for statistical modeling but using fewer parameters, leading to a more compact architecture. Further, no pooling layer and batch normalization layer are used in the final block. Since pooling with a kernel is trivial, it is omitted. Strides in both the convolutional and pooling layers are set to 1 to retain the feature map size. The output feature channels in the K-encoder are kept at 32, , (see blue cubes in Figure 2). Then, the decoded map is used to recover the scene radiance according to Eq. (4) (see the yellow circle in Figure 2). This structure is denoted FAMED-Net-SS, where “SS” stands for single scale.
We use the L2 loss to supervise the network during training:
where is the estimated scene radiance, represents learnable parameters of the network, and is the weight decay factor in the regularization term.
Iii-C The Multi-scale Variants of FAMED-Net: FAMED-Net-GP and FAMED-Net-LP
Objects at distinct distances are of different size in the captured images, leading to variably sized homogenous regions in the transmission map or map. To handle the multi-scale characteristics, we extend the proposed network to multi-scale by adopting a Gaussian pyramid architecture as shown in Figure 3(a). We down-sample the input hazy image to another two scales, , 1/2 scale and 1/4 scale, respectively. Then, we construct a K-encoder for each scale without sharing weights. Further, the estimated maps from the coarse scales are interpolated to the original scale and concatenated as:
where denote the interpolated maps. Bilinear interpolation is used for both down-sampling and up-sampling. Then, we introduce a fusion module to fuse the multi-scale estimates into a more reliable one, which is again implemented by a convolutional layer and a ReLU layer as:
Finally, is used to recover the scene radiance according to Eq. (4). This structure is denoted FAMED-Net-GP, where “GP” stands for Gaussian pyramid.
The L2 loss is used to supervise the network:
where and represent the ground truth and the estimated scene radiance at each scale, and represents the estimated scene radiance from the fusion module. and are loss weights, which are set to 1.
In addition to the Gaussian pyramid architecture, we also adopt a Laplacian pyramid architecture for comparison. As shown in Figure 3(b), the estimated map at the coarse scale is interpolated and added to the K-encoder output at the finer scale. Mathematically, it can be formulated as:
Therefore, it enforces the K-encoder at the finer scale to learn a residual . The other parts are kept the same as the Gaussian pyramid one. This structure is denoted FAMED-Net-LP, where “LP” stands for Laplacian pyramid. It is noteworthy that the receptive field of FAMED-Net-SS is which is similar to the local window size in prior-based dehazing methods, , in DCP  and MRP . As for FAMED-Net-GP and FAMED-Net-LP, their receptive fields become larger, , , which enables the network to learn more effective statistical regularities.
Iii-D Model Complexity Analysis
|Complexity111Evaluated with FLOPs, the number of floating-point multiplication-adds.||9.39x10|
The details of FAMED-Net are shown in Table I. It can be seen that FAMED-Net is very lightweight and compact thanks to the fully point-wise convolutions. For example, FAMED-Net-SS only contains 5,987 learnable parameters and has 9.39x10 FLOPs. The number of learnable parameters increases threefold in FAMED-Net-GP, while the FLOPs only increase by about 30%. FAMED-Net can process hazy images of arbitrary size due to its fully convolutional structure, with the computational cost increases linearly with the image size.
To reduce the required FLOPs for large images, we propose a fixed size testing strategy. First, we resize the hazy image with the longest side to 360 and input it into the network. Then, we resize the estimated map from the fusion module back to the original size using bilinear interpolation. Further, we use the fast-guided filter  to refine the interpolated map. The fast-guided filter is -times faster than the original -guided filter , with almost no visible degradation, where is the down-sampling ratio (refer to  for details). Finally, the scene radiance is recovered according to Eq.(4). In this way, we can process hazy images of arbitrary size at an almost fixed computational cost. We present our comparisons with state-of-the-art models in Table II including parameters, model size, and runtime. These comparisons clearly show that FAMED-Net is lightweight and computationally efficient. More details can be found in Section IV-C5.
|DCPDN222The number was calculated on 512x512 images since DCPDN required a fixed-size input. ||6.69x||255.6Mb||Pytorch(G)||0.0417|
To evaluate the performance of FAMED-Net, we compared it with state-of-the-art image prior-based methods including DCP , FVR , BCCR , GRM , CAP , and NLD  and deep CNN-based methods including DehazeNet , MSCNN , AOD-Net [20, 2], FPCNet , GFN , and DCPDN . We adopted the recently proposed RESIDE  as the benchmark dataset due to its large scale and diverse data sources and image contents. RESIDE contains 110,500 synthetic hazy indoor images (ITS) and 313,950 synthetic hazy outdoor images (OTS) in the training set. We reported the PSNR and SSIM for each method on the SOTS test set, which includes both indoor and outdoor scenes (500 of each). We also compared the subjective visual effects on real-world hazy images used in the literature. Ablation studies were conducted on TestSet-S containing 400 hazy indoor/outdoor images, a dataset initially used in a challenge .
FAMED-Net was trained for a total of 400,000 iterations on the combination of ITS and OTS in RESIDE. 128x128 patches randomly cropped from training images were used for training. Figure 4 shows the corresponding statistics of depth levels within the training patches. We quantized depth maps into 10 uniform levels according to the maximum and minimum depth values. Then, we counted the number of unique depth levels within each patch and calculated the histogram and its corresponding cumulative distribution as shown in Figure 4. As can be seen, almost 65% patches cover at least 3 depth levels and more than 40% patches cover at least 4 depth levels. It is noteworthy that since the sizes of training images from different scenes are around , each patch could cover diverse scene structures as evident by the statistics. Consequently, there are different levels of haze in each patch, , light and dense haze. It facilitates FAMED-Net with a receptive field of to learn effective feature representation while avoiding overfitting plain structures.
Hyper-parameters were tuned on the validation set. The batch size was set to 48. The initial learning rate was set to 0.00001, which decreased by 10 after 200,000 and 320,000 iterations. The momentum and weight decay were set to 0.9 and 0.0001, respectively. Average pooling was used unless otherwise specified. During testing, the kernel radius of the fast-guided filter was set to 48. The regularization parameter epsilon was set to 0.0001. The down-sampling factor was set to 4. FAMED-Net was implemented in Caffe  and run on a workstation with a 3.5 GHz CPU, 32G RAM, and Nvidia Titan XP GPUs.
Iv-a Ablation Experiments
|Model||3x3 Conv.||Training Data||Iterations||Indoor||Outdoor||Average||Indoor||Outdoor||Average|
Iv-A1 Ablations on the Basic Architecture
First, we conducted ablations on the components of the basic FAMED-Net architecture. We sampled a total of 40,000 images from ITS and OTS evenly to form a training set for ablations. Moreover, the models were trained in a total of 100,000 iterations. The learning rate decreased by 0.1 after 50,000 and 80,000 iterations. All other parameters were as described above. The results on TestSet-S are listed in Table III.
The dehazing results of FAMED-Net-FD4 with batch normalization were much better than FAMED-Net-NoBN. FAMED-Net-FD4 was also found to converge faster than FAMED-Net-NoBN. We also show the impact of the number of convolutional feature channels on the dehazing results. With more channels, the model tended to have a stronger representational capacity and achieved higher PSNR and SSIM scores. For example, FAMED-Net-S achieved a gain of 0.3 dB and 0.024 SSIM score over FAMED-Net-FD4 and a gain of 1.5 dB and 0.06 SSIM score over FAMED-Net-NoBN. With respect to the multi-scale architecture, with an additional down-scale branch, the PSNR score was improved by 0.2 dB but the SSIM score only decreased marginally. With all three scales, FAMED-Net-GP was the best architecture. Finally, we increased the feature channels in FAMED-Net-GP, but this only marginally improved the PSNR score and decreased the SSIM score. As a trade-off between accuracy and complexity, we chose FAMED-Net-GP as the representative architecture.
Iv-A2 Ablations on Training Data Volume and Training Iterations
We next investigated the impact of training data volume and training iterations. Specifically, we trained FAMED-Net-GP with 400,000 iterations and all the images in ITS and OTS, , a total of 424,450 images. The results are listed in Table IV. It can be seen that with sufficient training, FAMED-Net-GP improved. Moreover, the PSNR and SSIM significantly improved when FAMED-Net-GP was trained with all the images, producing a gain of 2.14 dB and 0.0425 SSIM score. Therefore, more training data benefits the deep neural network by exploiting its powerful representation capacity.
Iv-B Variants of the Multi-scale Architecture
Iv-B1 Additional 3x3 Convolutions for Learning Structural Features
Due to the fully point-wise convolutional structure, FAMED-Net-GP has limited ability on learn structural features. To see whether additional structural features benefit dehazing, we inserted additional 3x3 convolutional layers at the beginning of each scale in FAMED-Net-GP (denoted FAMED-Net-GP-3x3). We tested different feature channel configurations including 4 and 8. The results are shown in the first three rows in Table V.
Compared with FAMED-Net-GP (see the first and last rows in Table IV), FAMED-Net-GP-3x3 performed better with the same training settings. With more 3x3 convolutional channels, FAMED-Net-GP-3x3 trained with all training images was the best architecture, , 25.94 dB and 0.9180 SSIM score. Compared with its counterpart without 3x3 convolutional layers, gains of 0.26 dB and 0.01 SSIM score were achieved. However, this came at the cost of additional 6.69% parameters (, 1152) and 6.66% FLOPs (, 8.26x10).
Iv-B2 Laplacian Pyramid Architectures
In Section III-C, we also presented a Laplacian pyramid architecture FAMED-Net-LP (see Figure 3(b)). Compared with the Gaussian pyramid architecture FAMED-Net-GP (see the last row in Table V), FAMED-Net-LP achieved a marginally lower PSNR and a marginally higher SSIM. Generally, its performance was comparable to FAMED-Net-GP. Since there was no evident benefit to using residual learning, FAMED-Net-GP was used as our default multi-scale architecture in the following experiments.
Iv-B3 The Effectiveness of Max Pooling
For dehazing, effective local features are usually extracted from extreme pixel values including the dark channel (the minimum value of all the channels within a local patch) , local max contrast and saturation , and the learned features using the maxout operation in DehazeNet . Inspired by these studies, we hypothesized that max pooling may be more effective for aggregating local statistics and learning effective features for dehazing. To verify this hypothesis, we changed the average pooling operations in all the pooling layers to max pooling. This structure is denoted FAMED-Net-GP-MaxP and it was trained using the same settings as FAMED-Net-GP. The results are shown in the last row in Table V.
Compared with its counterpart using average pooling (last row in Table IV), FAMED-Net-GP-MaxP achieved a significant gain of 0.83 dB and 0.0091 SSIM score. It also outperformed FAMED-Net-GP-3x3 by 0.57 dB and achieved almost the same SSIM score. Therefore, we chose FAMED-Net-GP-MaxP as the representative model of the proposed architectures due to its light weight (a total of 17,991 parameters) and computational efficiency (1.24x10 FLOPs). For simplicity, it is denoted FAMED-Net in the following sections.
Iv-C Comparison with State-of-the-art Methods
To evaluate the performance of FAMED-Net, we compared it with several state-of-the-art methods including DCP , FVR , BCCR , GRM , CAP , NLD , DehazeNet , MSCNN , AOD-Net [20, 2], FPCNet , GFN  and DCPDN 
Iv-C1 Results on RESIDE SOTS
The PSNR and SSIM scores of the different methods are listed in Table VI. Several observations can be made. 1) CNN-based methods [18, 21, 20, 2, 22] generally outperformed the image prior-based methods [26, 45, 46, 47, 16, 17]. By learning features in a data-driven manner, CNN-based dehazing models had stronger representative capacities than image prior-based models, which are usually limited to specific scenarios. 2) CNN architecture matters. For example, FPCNet achieved a significant gain over its counterpart DehazeNet by using a lightweight, fully point-wise convolutional architecture. It achieved the second best SSIM score and even outperformed some complicated networks like AOD-Net, GFN, and DCPDN. Further, by integrating the imaging model into the network architecture, the end-to-end AOD-Net recovered the target haze-free image with higher accuracy than the none end-to-end methods [18, 19]. 3) FAMED-Net was the best performing method. Moreover, it significantly improved the PSNR and SSIM scores. For example, FAMED-Net surpassed the second-best methods by a large margin of 3.6 dB and 0.05 SSIM score.
|FPCNet ||21.84 (20.92/22.75)||0.8872 (0.8729/0.9014)|
|AOD-Net* ||23.43 (20.68/26.18)||0.8747 (0.8229/0.9266)|
|DCPDN ||20.81 (19.13/22.49)||0.8378 (0.8191/0.8565)|
|FAMED-Net||27.01 (25.00/29.03)||0.9371 (0.9172/0.9570)|
After carefully dissecting the proposed architecture of FAMED-Net and comparing it with state-of-the-art architectures, we can make the following conclusions. First, point-wise convolution plays a key role in constructing a compact and lightweight dehazing network. Cascaded point-wise convolutional layers are very effective for tackling the ill-posed dehazing problem by aggregating local statistic-based features layer by layer. Second, modeling the dehazing task in an end-to-end manner is beneficial. Third, a carefully designed multi-scale architecture can handle scale variance in complex scenes while only minimally increasing the computational cost. Finally, re-using features via dense connections like [20, 23, 50] leads to a better and more compact model.
Iv-C2 Subjective Evaluation
Subjective comparison on synthetic hazy images are presented in Figure 5. Dehazed results of MSCNN  on indoor images have residual haze indicated by the red boxes. Besides, MSCNN tended to produce over-saturated results with color distortions as indicated by the red arrows. Similar phenomena can also be found in the results of AOD-Net . Although FPCNet  achieved better results, there are some haze residual and color distortions as well. Moreover, MSCNN and FPCNet produced noisy results due to the incorrectly estimated transmission in regions enclosed by the blue boxes. The proposed FAMED-Net successfully restores the clear images with higher color fidelity and less haze/noise residual. It demonstrates the fitting ability of FAMED-Net learned from synthetic training images.
Next, we present the results on real-world hazy images in Figure 6 to compare different methods’ generalization ability. Close-up views in the red rectangles are also presented. It can be seen that DCP, MSCNN, and AOD-Net tended to produce over-saturated results, especially in sky regions. MSCNN also exhibits color artifacts, making the dehazed results unrealistic (see the first two images). AOD-Net dehazed images appear dimmer than the others. DehazeNet achieved better results, but still produced some color artifacts (see the middle part of the first image and the bluish artifact in the second image). FPCNet outperformed DehazeNet but retained some haze.
Using some enhanced results as input and a fusion strategy, GFN generated visually better results. However, color distortions in the middle part of the first image and the over-saturated second image are visually unpleasant. DCPDN produced better dehazing results and brighter results. However, some details are missing due to the over-exposure-like artifacts. Generally, FAMED-Net produced better or at least comparable results to state-of-the-art methods, , clear details with fewer color artifacts and high-fidelity sky regions. We also compared image enhancement for anti-halation using different methods in the last row. FAMED-Net also produced visually pleasing results. More results can be found in the supplement.
Iv-C3 Cross-set Generalization
We also compared the cross-set generalization between FAMED-Net and two recently proposed methods, GFN and DCPDN. We used RESIDE SOTS and TestA in  as two test sets. We used the pre-trained models of all three methods and did not fine-tune them. The results are listed in Table VII. It can be seen that FAMED-Net shows better generalization than GFN and DCPDN, which we ascribe to using the large-scale training set and the effectiveness of the proposed architecture.
Iv-C4 Analysis on the Learned Latent Statistical Regularities
Image prior-based methods including DCP , CAP  and NLD  assume prior statistics on haze-free images, which are used to enforce statistical regularities on recovering the target dehazed results . The learning-based methods also learn latent statistical regularities [18, 20, 21]. For example, DehazeNet and FPCNet, which regress the transmission, should produce a transmission map of all 1s for a haze-free image. In other words, they should learn dark channel-like statistical priors, , . As for AOD-Net and FAMED-Net, they regress a latent variable K implicitly. For a haze-free image, the atmospheric light is usually assumed to be white, , . Therefore, the corresponding can be deduced as from Eq. (3). Also, it should be a map all of 1s, , , where is the mean across three channels.
To compare the learned statistical regularities of different methods, we collected 100 haze-free images (two examples are shown in the first column of Figure 7) . These images were resized such that the long side was 480 pixels and the short side ranged from 100 to 480 pixels. Then, we calculated the dark channel, , and within each local patch of size . Next, we split the range of pixel value into 20 uniform bin centers and counted the corresponding number of pixels belonging to each bin on all images. Finally, we plotted the histograms of dark channel, , and for DCP, FPCNet, AOD-Net, and FAMED-Net in Figure 8. FAMED-Net learned a much more effective statistical regularity than DCP, FPCNet, and AOD-Net. Besides, the statistics of AOD-Net are far from zero. In other words, the trained network implicitly assumes that there is haze that needs to be removed in haze-free images. Therefore, it leads to over-dehazed artifacts, as seen in the third column. This is consistent with the visual results in Figure 6.
Iv-C5 Runtime Analysis
Following , we further compared the runtime of different methods on the indoor images () in RESIDE SOTS. The results are listed in Table II in Section III-D. Results of the classical methods above the line and cGAN are from [3, 24]. Others are reported using our workstation and the code released by the authors. We report the runtime of network forward computation and the whole algorithm including fast-guided filter refinement for FPCNet and FAMED-Net, as shown in separate rows in Table II. The numbers before/after the slash denote the runtime in CPU/GPU mode, , C/G. FAMED-Net runs very fast and reaches 85 fps and 35 fps without/with fast-guided filter refinement. In addition, we also list the number of parameters and model size of each CNN model. Compared with the recently proposed GFN, cGAN, and DCPDN, FAMED-Net is much more compact and lightweight.
Iv-D Limitations and Discussions
As stated in Section II-B and demonstrated in Section IV-C4, the proposed FAMED-Net implicitly learns a locally statistical regularity for dehazing like many prior- and learn-based methods [26, 16, 18, 19, 21, 20]. Though FAMED-Net outperforms these methods by leveraging more efficient architecture, it still has some limitations. Some examples of transmission maps estimated by FAMED-Net are shown in the bottom row in Figure 9. As indicated by the blue polygons, the transmission in the sky regions is incorrect, leading to under-dehazed artifacts as shown in Figure 6. It may be solved by incorporating high-level semantics into the dehazing network. However, it comes to the “chicken and egg” dilemma between the low-level enhancement and high-level understanding of degraded images. We suppose that it could be solved by jointly modeling the two correlated problems in a unified framework, which we leave as future work.
Besides, as evident by the low-light enhancement experiments in the supplement and color constancy results in , point-wise convolutions could be used for statistical modeling of illumination, color cast, etc. Referring to the haze imaging model in , we will also exploit FAMED-Net’s potential for haze removal in the presence of non-uniform atmosphere light, , artificial ambient light in nighttime haze environment. Extending FAMED-Net to remove heterogeneous haze is also promising by investigating region-based techniques, , haze density-aware segmentation.
In this paper, we introduce a novel fast and accurate multi-scale end-to-end dehazing network called FAMED-Net to tackle the challenging single image dehazing problem. FAMED-Net comprises three encoders at different scales and a fusion module, which is able to efficiently learn the haze-free image directly. Each encoder consists of cascaded point-wise convolutional layers and pooling layers via a densely connected mechanism. By leveraging a fully point-wise structure, FAMED-Net is lightweight and computationally efficient. Extensive experiments on public benchmark datasets and real-world hazy images demonstrate the superiority of FAMED-Net over other top performing models: it is a fast, lightweight, and accurate deep architecture for single image dehazing.
-  B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “End-to-end united video dehazing and detection,” arXiv preprint arXiv:1709.03919, 2017.
-  Y. Liu, G. Zhao, B. Gong, Y. Li, R. Raj, N. Goel, S. Kesav, S. Gottimukkala, Z. Wang, W. Ren et al., “Improved techniques for learning to dehaze and beyond: A collective study,” arXiv preprint arXiv:1807.00202, 2018.
-  B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single image dehazing and beyond,” IEEE Transactions on Image Processing, 2018.
-  Z. Tu, X. Chen, A. L. Yuille, and S.-C. Zhu, “Image parsing: Unifying segmentation, detection, and recognition,” International Journal of computer vision, vol. 63, no. 2, pp. 113–140, 2005.
-  J.-P. Tarel, N. Hautiere, A. Cord, D. Gruyer, and H. Halmaoui, “Improved visibility of road scene images under heterogeneous fog,” in Intelligent Vehicles Symposium (IV), 2010 IEEE. Citeseer, 2010, pp. 478–485.
-  C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” International Journal of Computer Vision, pp. 1–20, 2018.
-  K. Tan and J. P. Oakley, “Enhancement of color images in poor visibility conditions.” in ICIP, vol. 2, 2000.
-  Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “Instant dehazing of images using polarization,” in Proc. Computer Vision & Pattern Recognition Vol, vol. 1, 2001, pp. 325–332.
-  S. K. Nayar and S. G. Narasimhan, “Vision in bad weather,” in The IEEE International Conference on Computer Vision, vol. 2. IEEE, 1999, pp. 820–827.
-  Q. Liu, X. Gao, L. He, and W. Lu, “Single image dehazing with depth-aware non-local total variation regularization,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 5178–5191, 2018.
-  A. Wang, W. Wang, J. Liu, and N. Gu, “Aipnet: Image-to-image single image dehazing with atmospheric illumination prior,” IEEE Transactions on Image Processing, 2018.
-  Z. Li and J. Zheng, “Single image de-hazing using globally guided image filtering,” IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 442–450, 2018.
-  R. Fattal, “Single image dehazing,” ACM transactions on graphics (TOG), vol. 27, no. 3, p. 72, 2008.
-  K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 12, pp. 2341–2353, 2011.
-  K. Tang, J. Yang, and J. Wang, “Investigating haze-relevant features in a learning framework for image dehazing,” in The IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2995–3000.
-  Q. Zhu, J. Mai, and L. Shao, “A fast single image haze removal algorithm using color attenuation prior,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3522–3533, 2015.
-  D. Berman, S. Avidan et al., “Non-local image dehazing,” in The IEEE conference on computer vision and pattern recognition, 2016, pp. 1674–1682.
-  B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187–5198, 2016.
-  W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in European Conference on Computer Vision. Springer, 2016, pp. 154–169.
-  B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “Aod-net: All-in-one dehazing network,” in The IEEE International Conference on Computer Vision, vol. 1, no. 4, 2017, p. 7.
-  J. Zhang, Y. Cao, Y. Wang, C. Wen, and C. W. Chen., “Fully point-wise convolutional neural network for modeling statistical regularities in natural images,” in ACM Multimedia Conference, 2018.
-  W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
-  R. Li, J. Pan, Z. Li, and J. Tang, “Single image dehazing via conditional generative adversarial network,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  T. M. Bui and W. Kim, “Single image dehazing using color ellipsoid prior,” IEEE Transactions on Image Processing, vol. 27, no. 2, pp. 999–1009, 2018.
-  K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” in The IEEE conference on computer vision and pattern recognition, 2009.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680.
-  E. J. McCartney, “Optics of the atmosphere: scattering by molecules and particles,” New York, John Wiley and Sons, Inc., 1976. 421 p., 1976.
-  S. G. Narasimhan and S. K. Nayar, “Vision and the atmosphere,” International Journal of Computer Vision, vol. 48, no. 3, pp. 233–254, 2002.
-  D. Yang and J. Sun, “Proximal dehaze-net: A prior learning-based deep network for single image dehazing,” in The European Conference on Computer Vision (ECCV), September 2018.
-  I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis,” IEEE transactions on information theory, vol. 36, no. 5, pp. 961–1005, 1990.
-  D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, no. 2, pp. 91–110, 2004.
-  W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in The IEEE conference on computer vision and pattern recognition, 2017, pp. 624–632.
-  R. Yu, W. Liu, Y. Zhang, Z. Qu, D. Zhao, and B. Zhang, “Deepexposure: Learning to expose photos with asynchronously reinforced adversarial learning,” in Advances in Neural Information Processing Systems, 2018, pp. 2149–2159.
-  E. L. Denton, S. Chintala, R. Fergus et al., “Deep generative image models using a laplacian pyramid of adversarial networks,” in Advances in neural information processing systems, 2015, pp. 1486–1494.
-  K. Xu, Z. Zhang, and F. Ren, “Lapran: A scalable laplacian pyramid reconstructive adversarial network for flexible compressive sensing reconstruction,” in The European Conference on Computer Vision (ECCV), 2018, pp. 485–500.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2018.
-  T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  S. Xie and Z. Tu, “Holistically-nested edge detection,” in The IEEE international conference on computer vision, 2015, pp. 1395–1403.
-  C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets,” in Artificial Intelligence and Statistics, 2015, pp. 562–570.
-  A. Hyvärinen, J. Hurri, and P. O. Hoyer, Natural Image Statistics. Springer-Verlag London, 2009.
-  K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 6, pp. 1397–1409, 2013.
-  J. Zhang, Y. Cao, S. Fang, Y. Kang, and C. W. Chen, “Fast haze removal for nighttime image using maximum reflectance prior,” in The IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7418–7426.
-  K. He and J. Sun, “Fast guided filter,” arXiv preprint arXiv:1505.00996, 2015.
-  J.-P. Tarel and N. Hautiere, “Fast visibility restoration from a single color or gray level image,” in 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009, pp. 2201–2208.
-  G. Meng, Y. Wang, J. Duan, S. Xiang, and C. Pan, “Efficient image dehazing with boundary constraint and contextual regularization,” in The IEEE international conference on computer vision, 2013, pp. 617–624.
-  C. Chen, M. N. Do, and J. Wang, “Robust image and video dehazing with visual artifact suppression via gradient residual minimization,” in European Conference on Computer Vision, 2016, pp. 576–591.
-  W. Ren, Z. Wang, Y. Guo, g. Meng, X. Fan, and J. Guo, “Chinamm18dehazing,” https://rwenqi.github.io/ChinaMM18dehazing/, 2018.
-  Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in The 22nd ACM international conference on Multimedia. ACM, 2014, pp. 675–678.
-  G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.” in CVPR, vol. 1, no. 2, 2017, p. 3.
Jing Zhang received the B.S. degree from the Henan University and the Ph.D from the University of Science and Technology of China (USTC) in 2010 and 2015, respectively. He used to work as a research fellow at the IFLYTEK Research. He joined the School of Automation in the Hangzhou Dianzi University as a lecturer in 2017. He was a visiting scholar at the University of Technology Sydney from 2018 to 2019. Currently, he is a research fellow at the UBTECH Sydney Artificial Intelligence Centre. His research interests include computer vision and multimedia. He published several papers at IEEE CVPR, ACM Multimedia, AAAI, IEEE TCSVT, Information Sciences, Neurocomputing, etc. He serves as a reviewer for a number of journals and conferences such as IJCV, TIP, TCB, TCSVT, Information Sciences, CVPR, ACM Multimedia.
Dacheng Tao(Fâ15) is Professor of Computer Science and ARC Laureate Fellow in the School of Computer Science and the Faculty of Engineering, and the Inaugural Director of the UBTECH Sydney Artificial Intelligence Centre, at The University of Sydney. He mainly applies statistics and mathematics to Artificial Intelligence and Data Science. His research results have expounded in one monograph and 200+ publications at prestigious journals and prominent conferences, such as IEEE T-PAMI, T-IP, T-SP, T-NNLS, T-CYB, IJCV, JMLR, NIPS, ICML, CVPR, ICCV, ECCV, ICDM, and KDD, with several best paper awards, such as the IEEE ICDMâ07 best theory/algorithm paper runner up award, the IEEE ICDMâ13 best student paper award, the 2014 ICDM 10-year highest-impact paper award, the 2017 IEEE Signal Processing Society Best Paper Award, and the 2018 IJCAI distinguished paper award. He received the 2015 Australian Scopus-Eureka Prize and the 2018 IEEE ICDM Research Contributions Award. He is a Fellow of the Australian Academy of Science, AAAS, IEEE, IAPR, OSA and SPIE.