Follow and star
Never get lost again
Computer Vision Research Institute
Public Account ID | ComputerVisionGzq
Learning Group | Scan the QR code to get the joining method on the homepage
In order to overcome the overfitting problem of the dehazing model trained on the synthetic data set, Many recent methods attempt to use unpaired data for training to improve the generalization ability of the model. However, most of these methods simply follow the ideas of CycleGAN to construct the dehazing cycle and fogging cycle, but ignore the physical characteristics of the haze environment in the real world, that is, the impact of haze on the visibility of objects changes with depth and fog density.
In this paper, we propose a self-enhanced image dehazing framework, called D4 (Dehazing via Decomposing transmission map into Density and Depth), for image dehazing and fog generation. Rather than simply estimating transmission maps or clear images, our proposed framework focuses on exploring the scattering coefficients and depth information in hazy and clear images. With the estimated scene depth, our method is able to re-render hazy images with different thicknesses of fog and used for data augmentation to train the dehazing network. It is worth noting that the entire training process successfully recovers the scattering coefficient, depth map and clear content from a single hazy image by relying only on unpaired hazy and clear images. Comprehensive experiments on
show that our method outperforms state-of-the-art non-pairwise dehazing methods with fewer parameters and fewer FLOPs. This work was completed by Jingdong Discovery Research Institute in conjunction with Tianjin University and Sydney University , and has been accepted by CVPR2022.
1 Research background
Haze is a natural phenomenon caused by the scattering effect of aerosol particles in the atmosphere. It can seriously affect the visibility of content in images, affecting both humans and computer vision systems.
relies on the powerful learning capabilities of deep neural networks, and a large number of supervised methods have been proposed and applied to image defogging. By training using a large number of synthetic hazy-clear image pairs, the supervised deep dehazing method achieves satisfactory results on a specific test set. However, there is a large gap between synthetic hazy images and real-world hazy images. Dehazing models trained solely on pairs of images can easily overfit, resulting in poor generalization to real-world hazy images.
Since hazy/clear images are difficult to obtain in the real world, in recent years, researchers have proposed many deep learning methods using non-paired hazy/clear images to train image dehazing models. Among them, many methods use ideas based on CycleGAN [1] to construct dehazing cycles and fogging cycles, so that content consistency can be maintained while converting hazy images and clear images.
However, we believe that simply using the idea of CycleGAN to realize the conversion between the foggy image domain and the clear image domain end-to-end through the network cannot well solve the problem of non-paired image dehazing. Existing construction loop-based dehazing methods ignore the physical characteristics of real foggy environments, that is, the impact of real-world fog on images changes with changes in fog concentration and depth. This relationship has been determined by atmospheric scattering Model [2] gives a description, that is, a hazy image can be expressed as:
(1)
where J(x) is a clear image and A is atmospheric light, which can be directly determined using the method in [3]. t(x) is the transmission image, which can be further expressed as:
(2)
where
The goal of this method is to introduce a physical model that considers fog density and scene depth based on the original CycleGAN processing non-paired image dehazing method. This allows the model to synthesize more realistic fog with varying thickness during the training process, thereby achieving data enhancement and thereby improving the model's dehazing effect.
Figure 1 (1) Illustration of non-pairwise defogging based on CycleGAN, (2) Illustration of the proposed method and (3) result comparison
2 Self-enhanced non-pairwise image defogging method based on depth and density decomposition
Pairwise image defogging method
We propose a self-enhancing non-pairwise image dehazing framework based on depth and fog density decomposition. The training process includes two branches, the dehazing-fogging branch and the fogging-dehazing branch. As shown in the upper part of Figure 2, in the dehazing-fogging branch, a hazy image is first input into the dehazing network to obtain the estimated transmission map and estimated scattering coefficient , and further synthesizes a clear image through Equation (1).
At the same time, according to equation (2), its depth can be calculated through the estimated transmission map and scattering coefficient. Then input into the depth estimation network to get the estimated depth map. Then the obtained depth map and the previously obtained scattering coefficient are used to obtain the coarse foggy image according to equations (1) and (2), and then the final foggy image is obtained through the refinement network. In the fogging-dehazing branch, as shown in the lower half of Figure 2, the starting point becomes a clear image.
first inputs the depth estimation network to obtain the estimated depth. Combined with the scattering factors randomly sampled in a uniform distribution, a coarse fogged image is obtained according to formulas (1) and (2), and then the fogged image is obtained through the refinement network. The obtained hazy image is then passed through the defogging network to obtain the estimated transmission map and estimated scattering coefficient , and a clear image is further synthesized through equation (1).
Random sampling of scattering factors is one of our innovations, because fog in nature is divided into light and heavy, so by randomly sampling the scattering factors and inputting them into the fog synthesis part below, our network can During the training process, mist with rich thickness changes is provided to achieve the purpose of self-enhancement.
Figure 2 Schematic diagram of the framework training process
Note here the clear/fog images paired with yes and no. In order to ensure that the entire framework can be trained normally, we use several loss functions, including cycle consistency loss, adversarial loss, pseudo-scattering factor supervised loss and pseudo-depth supervised loss.
cycle consistency loss requires that in the two branches, the reconstructed foggy image should be consistent with the given foggy image, and the reconstructed clear image should be consistent with the given clear image. Its purpose is to maintain consistency in image content. Cycle consistency loss is expressed as:
The adversarial loss evaluates whether the generated image belongs to a specific domain. In other words, it constrains that our dehazed and rehazed images should be visually realistic and follow the same distribution as the images in the training set and , respectively. For the dehazing network and the corresponding discriminator , the adversarial loss can be expressed as:
where is a real clear image sample sampled from the clear image collection. It is the defogging result obtained through the defogging network. is a discriminator used to determine whether the input image belongs to the clear domain. Correspondingly, the adversarial loss used by the image refinement network and the corresponding discriminator can be expressed as:
where is a real hazy image sample sampled from the hazy image collection. It is a foggy image obtained by thinning the network. Is a discriminator used to determine whether the input image belongs to the foggy domain. Since there is no directly available pairwise depth information and paired scattering factor information,
is used to train the depth estimation network and the scattering factor estimation network. We introduce pseudo-scattering factor supervised loss and pseudo-depth supervised loss to train these two sub-networks.
pseudo-scattering factor supervision loss means that in the fogging-dehazing branch, the scattering factor predicted by the dehazing network should be consistent with the randomly generated value. It can be expressed as:
Pseudo-depth supervision loss means that in the defogging-fogging branch, the depth predicted by the deep network should be consistent with that calculated by and .It can be expressed as:
where the depth estimation network is directly optimized by the depth estimation loss, and the remaining modules are optimized by
3 Experimental results
Table 1 Performance of each method on each data set
We compare the proposed method with other supervised, Unsupervised and unpaired defogging methods were compared. The supervised comparison methods include EPDN [4], FFANet [5], HardGAN [6], PSD [7], and the unpaired methods include CycleGAN [1] , CycleDehaze[8], DisentGAN[9], RefineDNet[10], unsupervised methods include DCP[3], YOLY[11]. Quantitative experimental results of
are compared with . In order to verify that our method has better generalization performance compared to supervised methods and better dehazing performance compared to other unsupervised or unpaired methods, we tested these methods on the SOTS-indoor dataset. Train and test their performance on other datasets. At the same time, we also tested the model parameter quantities and FLOPs of these methods to test the efficiency of these models. The results are shown in Table 1. Qualitative experimental results of
compared to . In order to verify the advantages of our method compared with other methods, we also conducted qualitative tests on multiple data sets and real hazy images. The results are shown in Figures 3 and 4. Among them, the first group and the second group of images in Figure 3 are the test set of SOTS-indoor. The distribution is similar to the training set. It can be seen that FFANet has the best dehazing effect, and our method is better than the other methods except FFANet. The third and fourth pictures of
are from the SOTS-outdoor and IHAZE data sets respectively, and have different distributions from the training set. It can be seen that our method is more thorough in dehazing than other methods, and has less color distortion than other methods such as cycledehaze, and the generated results are more natural. Figure 4 shows two examples of real image dehazing. It can be seen that the dehazing results of our method are significantly better than other methods, indicating that the generalization ability of our model has obvious advantages over other models.
In addition, our method can also be used to generate foggy images. This type of technology can be applied to image or video editing. Compared with other methods, the foggy images generated by our method can change the fog at will. thickness, and more realistic, as shown in Figure 5.
In addition, unlike other non-paired image dehazing methods, our model also supports relative depth prediction for clear images. The effect is shown in Figure 6. Although compared with other supervised depth estimation networks, the depth estimation The accuracy is limited, but our method is the first to be able to estimate scene depth using unpaired hazy/clear images trained on it.
Figure 3 Comparison of the qualitative effects of each method on the test set
Figure 4 Comparison of the dehazing results of each method on real images
Figure 5 The effect of the proposed method on generating hazy images
Figure 6 The proposed method is Effect on Depth Estimation
4
Conclusion
This paper proposes a self-enhanced non-pairwise image dehazing framework D4, which decomposes the estimation of the transmission map into predictions of fog density (scattering factor) and depth map. Based on the estimated depth, our method is able to re-render hazy images with different fog thicknesses and serve as self-augmentation to improve model dehazing performance. Sufficient experiments verify the superiority of our method over other defogging methods.
However, our method also has limitations. It usually overestimates the transmission map of extremely bright areas, which will mislead the depth estimation network to predict smaller depth values for overly bright areas. And we found that low-quality training data can lead to unstable training. Nonetheless, our proposed idea of further decomposition of variables in physical models can be extended to other tasks, such as low-light enhancement. We hope that our approach can inspire future work, especially on non-pairwise learning tasks in low-level vision.
article: https://openaccess.thecvf.com/content/CVPR2022/html/Yang_Self-Augmented_Unpaired_Image_Dehazing_via_Density_and_Depth_Decomposition_CVPR_2022_paper.html
code: The code has been published https://github.com/YaN9-Y/D4
Reference
[1] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In ICCV, pages 2223–2232, 2017
[2] Srinivasa G Narasimhan and Shree K Nayar. Chromatic framework for vision in bad weather. In CVPR, volume 1, pages 598–605, 2000.
[3] Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. IEEE TPAMI, 33(12) ):2341–2353, 2010.
[4] Yanyun Qu, Yizi Chen, Jingying Huang, and Yuan Xie. En-hanced pix2pix dehazing network. In CVPR, pages 8160– 8168, 2019.
[5] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In AAAI, volume 34, pages 11908– 11915, 2020
[6] Qili Deng, Ziling Huang, Chung-Chi Tsai , and Chia-Wen Lin. Hardgan: A haze-aware representation distillation gan for single image synthetic dehazing. In ECCV, pages 722–738.
[7] Zeyuan Chen, Yangchao Wang, Yang Yang, and Dong Liu. Psd: Principled -to-real dehazing guided by phys- ical priors. In CVPR, pages 7180–7189, June 2021. [8] Shiwei Shen, Guoqing Jin, Ke Gao, and Yongdong Zhang. Ape-gan: Adversarial perturbation elimination with gan. arXiv preprint arXiv:1707.05474, 2017.
[8] Deniz Engin, Anil Genc¸, and Hazim Kemal Ekenel. Cycle-dehaze: Enhanced cyclegan for single image dehazing. In CVPRW, pages 825–833, 2018.
[9] Xitong Yang , Zheng Xu, and Jiebo Luo. Towards percep-tual image dehazing by physics-based disentanglement and adversarial training. In AAAI, volume 32, pages 7485–7492, 2018.
[10] Shiyu Zhao, Lin Zhang, Ying Shen, and Yicong Zhou. Refinednet: A weakly supervised refinement framework for single image dehazing. IEEE TIP, 30:3391–3404, 2021.
[11] Boyun Li, Yuanbiao Gou, Shuhang Gu, Jerry Zitao Liu, Joey Tianyi Zhou, and Xi Peng. You only look yourself: Unsupervised and untrained single image dehazing neural net- work. IJCV, 129(5):1754–1767, 2021.
© THE END
Please contact this public account for authorization for reprinting
Computer Vision Research Institute The study group is waiting for you to join!
ABOUT
Computer Vision Research Institute
Computer Vision Research Institute mainly involves the field of deep learning, and is mainly committed to face detection, face recognition, multi-target detection, target tracking, image segmentation and other research directions. The institute will continue to share the latest new paper algorithm frameworks in the future. The difference in our reform this time is that we must focus on "research." After that, we will share the practical process in the corresponding fields, so that everyone can truly experience the real scenario of getting rid of the theory, and cultivate the habit of loving hands-on programming and thinking with their brains!
VX:2311123606
