KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Lightweight multiple scale-patch dehazing network for real-world hazy image

  • Wang, Juan (Hubei Energy Internet Engineering Technology Research Center, Hubei University of Technology) ;
  • Ding, Chang (Hubei Energy Internet Engineering Technology Research Center, Hubei University of Technology) ;
  • Wu, Minghu (Hubei Energy Internet Engineering Technology Research Center, Hubei University of Technology) ;
  • Liu, Yuanyuan (Hubei Energy Internet Engineering Technology Research Center, Hubei University of Technology) ;
  • Chen, Guanhai (Hubei Energy Internet Engineering Technology Research Center, Hubei University of Technology)
  • Received : 2021.08.20
  • Accepted : 2021.11.27
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Image dehazing is an ill-posed problem which is far from being solved. Traditional image dehazing methods often yield mediocre effects and possess substandard processing speed, while modern deep learning methods perform best only in certain datasets. The haze removal effect when processed by said methods is unsatisfactory, meaning the generalization performance fails to meet the requirements. Concurrently, due to the limited processing speed, most dehazing algorithms cannot be employed in the industry. To alleviate said problems, a lightweight fast dehazing network based on a multiple scale-patch framework (MSP) is proposed in the present paper. Firstly, the multi-scale structure is employed as the backbone network and the multi-patch structure as the supplementary network. Dehazing through a single network causes problems, such as loss of object details and color in some image areas, the multi-patch structure was employed for MSP as an information supplement. In the algorithm image processing module, the image is segmented up and down for processed separately. Secondly, MSP generates a clear dehazing effect and significant robustness when targeting real-world homogeneous and nonhomogeneous hazy maps and different datasets. Compared with existing dehazing methods, MSP demonstrated a fast inference speed and the feasibility of real-time processing. The overall size and model parameters of the entire dehazing model are 20.75M and 6.8M, and the processing time for the single image is 0.026s. Experiments on NTIRE 2018 and NTIRE 2020 demonstrate that MSP can achieve superior performance among the state-of-the-art methods, such as PSNR, SSIM, LPIPS, and individual subjective evaluation.

Keywords

References

  1. Mccartney, E. J., "Scattering phenomena. (book reviews: Optics of the atmosphere. scattering by molecules and particles)," Science, 196, 1084-1085, 1977. https://doi.org/10.1126/science.196.4294.1084.b
  2. K. He, J. Sun and X. Tang, "Single Image Haze Removal Using Dark Channel Prior," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2341-2353, Dec. 2011. https://doi.org/10.1109/TPAMI.2010.168
  3. R. T. Tan, "Visibility in bad weather from a single image," in Proc. of 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008.
  4. Q. Zhu, J. Mai and L. Shao, "A Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior," IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3522-3533, Nov. 2015. https://doi.org/10.1109/TIP.2015.2446191
  5. B. Cai, X. Xu, K. Jia, C. Qing and D. Tao, "DehazeNet: An End-to-End System for Single Image Haze Removal," IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187-5198, Nov. 2016. https://doi.org/10.1109/TIP.2016.2598681
  6. A Benoit, Leonel Cuevas, Jean-Baptiste Thomas, "Deep learning for dehazing: Comparison and analysis," in Proc. of Colour and Visual Computing Symposium (CVCS), 2018.
  7. B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, "AOD-Net: All-in-one dehazing network," in Proc. of the IEEE International Conference on Computer Vision, vol. 1, p. 7, 2017.
  8. Zhang, H., Patel, V. M., Patel, V. M. and Patel, V. M, "Densely connected pyramid dehazing network," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 3194-3203, 2018.
  9. X.H. Liu, Y.R. Ma, Z.H. Shi and J. Chen, "GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing," in Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7314-7323, 2019.
  10. W.Q. Ren, L. Ma, J.W. Zhang, J.S. Pan, X.C. Cao, W. Liu, M.H. Yang, "Gated Fusion Network for Single Image Dehazing," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3253-3261, 2018.
  11. H. Dong, J.S. Pan, L. Xiang, Z. Hu, X.Y. Zhang, F. Wang, M.H. Yang, "Multi-Scale Boosted Dehazing Network With Dense Feature Fusion," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2154-2167, 2020.
  12. Y. Dong, Y. Liu, H. Zhang, S. Chen, Y. Qiao, "FD-GAN: Generative Adversarial Networks with Fusion-discriminator for Single Image Dehazing," in Proc. of the AAAI Conference on Artificial Intelligence, vol. 34(07), pp. 10729-10736. 2020.
  13. H. Zhu, X. Peng, V. Chandrasekhar, L. Li, J.H. Lim, "DehazeGAN: When Image Dehazing Meets Differential Programming," IJCAI, 2018.
  14. W. Wang, A. Wang, Q. Ai, C. Liu and J. Liu, "AAGAN: Enhanced Single Image Dehazing With Attention-to-Attention Generative Adversarial Network," IEEE Access, vol. 7, pp. 173485-173498, 2019. https://doi.org/10.1109/access.2019.2957057
  15. N. Wang, Y.B. Zhou, F.L. Han, H.Y. Zhu, Y.J. Zheng, "UWGANUnderwater GAN for Real-world Underwater Color," arXiv:1912.10269. 2019.
  16. J. Zhang, Y. Cao, and Z.F. Wang, "Nighttime haze removal based on a new imaging model," in Proc. of 2014 IEEE International Conference on Image Processing (ICIP), pp. 4557-4561, 2014.
  17. Y. Li, R. T. Tan, and M. S. Brown, "Nighttime haze removal with glow and multiple light colors," in Proc. of the IEEE International Conference on Computer Vision (ICCV), pp. 226-234, 2015.
  18. J. Zhang, Y. Cao, S. Fang, Y. Kang, and C. W. Chen, "Fast haze removal for nighttime image using maximum reflectance prior," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7418-7426, 2017.
  19. J. Zhang, Y. Cao, Z. J. Zha, D. C. Tao, "Nighttime Dehazing with a Synthetic Benchmark," in Proc. of the 28th ACM International Conference on Multimedia, 2020.
  20. S. D. Das, S.t Dutta, "Fast Deep Multi-Patch Hierarchical Network for Nonhomogeneous Image Dehazing," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 482-483, 2020.
  21. T. T. Guo, X. L. Li, V. Cherukuri, and V. Monga, "Dense scene information estimation network for dehazing," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2122-2130, 2019.
  22. T. T. Guo, V. Cherukuri, and V. Monga, "Dense '123' color enhancement dehazing network," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2131-2139, 2019.
  23. X. Qin, Z. L. Wang, Y. C. Bai, X. D. Xie, and H. Z.Jia, "FFA-Net: Feature fusion attention network for single image dehazing," in Proc. of the AAAI Conference on Artificial Intelligence, vol. 34(07), pp. 11908-11915, 2020.
  24. R. T. Li, X. Y. Zhang, S. D. You, Y. Li, "Learning to Dehaze From Realistic Scene with A Fast Physics Based Dehazing Network," arXiv preprint arXiv:2004.08554, 2020.
  25. Yang, X., Xu, Z., Luo, J., "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training," in Proc. of the AAAI Conference on Artificial Intelligence, vol. 32(1), 2018.
  26. D. Chen et al., "Gated Context Aggregation Network for Image Dehazing and Deraining," in Proc. of 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1375-1383, 2019.
  27. Ronneberger O., Fischer P., Brox T, "U-Net: Convolutional Networks for Biomedical Image Segmentation," In: Navab N., Hornegger J., Wells W., Frangi A. (eds) Medical Image Computing and Computer-Assisted Intervention (MICCAI). vol 9351, pp. 234-241, 2015.
  28. D. Engin, A. Genc, H. K. Ekenel, "Cycle-Dehaze: Enhanced CycleGAN for Single Image Dehazing," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 825-833, 2018.
  29. J. Y. Zhu, T. Park, P. Isola, A. A. Efros, "Unpaired Image-To-Image Translation Using CycleConsistent Adversarial Networks," in Proc. of the IEEE International Conference on Computer Vision (ICCV), pp. 2242 - 2251, 2017.
  30. Y. Y. Qu, Y. Z. Chen, J. Y. Huang, Y. Xie, "Enhanced Pix2pix Dehazing Network," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8152-8160, 2019.
  31. A. Mehta, H. Sinha, P. Narang, M. Mandal, "HIDEGAN: A Hyperspectral-guided Image Dehazing GAN," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 846-856, 2020.
  32. Y.J. Shao, L. Li, W. Q. Ren, C. X. Gao, N. Sang, "Domain Adaptation for Image Dehazing," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2805-2814, 2020.
  33. Y. K. Yu, H. Liu, M. H. Fu, J. Chen, X. Y. Wang, K. Y. Wang, "A Two-Branch Neural Network for Non-Homogeneous Dehazing via Ensemble Learning," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 193-202, 2021.
  34. H. Y. Wu, Y. Y. Qu, S. H. Lin, J. Zhou, R. Z. Qiao, Z. Z. Zhang, Y. Xie, L. Z. Ma, "Contrastive Learning for Compact Single Image Dehazing," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10551-10560, 2021.
  35. S. Zhao, L. Zhang, Y. Shen and Y. Zhou, "RefineDNet: A Weakly Supervised Refinement Framework for Single Image Dehazing," in Proc. of IEEE Transactions on Image Processing (TIP), vol. 30, pp. 3391-3404, 2021.
  36. H. G. Zhang, Y. C. Dai, H. D. Li, and P. Koniusz, "Deep stacked hierarchical multi-patch network for image deblurring," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5971-5979, 2019.
  37. Ancuti, C., Ancuti, C. O., and Timofte, R., "Ntire 2018 challenge on imagedehazing: Methods and results," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 891-901, 2018.
  38. C.O. Ancuti, C. Ancuti, R. Timofte, "NH-HAZE: An Image Dehazing Benchmark with Non-Homogeneous Hazy and Haze-Free Images," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1798-1805, 2020.
  39. B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, "Benchmarking single-image dehazing and beyond," IEEE Transactions on Image Processing (TIP), vol. 28(1), pp. 492-505, 2019. https://doi.org/10.1109/tip.2018.2867951
  40. C. Ancuti, C. O. Ancuti, and C. De Vleeschouwer, "D-HAZY: a dataset to evaluate quantitatively dehazing algorithms," in Proc. of IEEE International Conference on Image Processing (ICIP), IEEE, pp. 2226-2230, 2016.
  41. Ancuti C., Ancuti C.O., Timofte R., De Vleeschouwer C, "I-HAZE: a dehazing benchmark with real hazy and haze-free indoor images," in Proc. of International Conference on Advanced Concepts for Intelligent Vision Systems, Springer, Cham, 2018.
  42. C. O. Ancuti, C. Ancuti, R. Timofte, C.D. Vleeschouwer, "O-HAZE: a dehazing benchmark with real hazy and haze-free outdoor images," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 754-762, 2016.
  43. Zhang, Y. F., Li D., and G. Sharma, "Hazerd: an outdoor scene dataset and benchmark for single image dehazing," in Proc. of 2017 IEEE International Conference on Image Processing (ICIP), IEEE, pp. 3205-3209, 2017.
  44. A. Gaidon, Q. Wang, Y. Cabon, E. Vig, "Virtual worlds as proxy for multi-object tracking analysis," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4340-4349, 2016.
  45. S. Zhao, L. Zhang, S. Huang, Y. Shen and S. Zhao, "Dehazing Evaluation: Real-World Benchmark Datasets, Criteria, and Baselines," IEEE Transactions on Image Processing, vol. 29, pp. 6947-6962, 2020. https://doi.org/10.1109/tip.2020.2995264
  46. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, "The unreasonable effectiveness of deep features as a perceptual metric," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018.