Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Classification and Restoration of Compositely Degraded Images using Deep Learning

딥러닝 기반의 복합 열화 영상 분류 및 복원 기법

  • Yun, Jung Un (Department of Information and Communication Engineering, Inha University) ;
  • Nagahara, Hajime (Institute for Datability Science, Osaka University) ;
  • Park, In Kyu (Department of Information and Communication Engineering, Inha University)
  • Received : 2019.04.04
  • Accepted : 2019.05.02
  • Published : 2019.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

The CNN (convolutional neural network) based single degradation restoration method shows outstanding performance yet is tailored on solving a specific degradation type. In this paper, we present an algorithm of multi-degradation classification and restoration. We utilize the CNN based algorithm for solving image degradation classification problem using pre-trained Inception-v3 network. In addition, we use the existing CNN based algorithms for solving particular image degradation problems. We identity the restoration order of multi-degraded images empirically and compare with the non-reference image quality assessment score based on CNN. We use the restoration order to implement the algorithm. The experimental results show that the proposed algorithm can solve multi-degradation problem.

CNN (convolutional neural network) 기반의 단일 열화 영상 복원 방법은 우수한 성능을 나타내지만 한가지의 특정 열화를 해결하는 데 맞춤화 되어있다. 본 연구에서는 복합적으로 열화 된 영상 분류 및 복원을 위한 알고리즘을 제시한다. 복합 열화 영상 분류 문제를 해결하기 위해 CNN 기반의 알고리즘인 사전 학습된 Inception-v3 네트워크를 활용하고, 영상 열화 복원을 위해 기존의 CNN 기반의 복원 알고리즘을 사용하여 툴체인을 구성한다. 실험적으로 복합 열화 영상의 복원 순서를 추정하였으며, CNN 기반의 영상 화질 측정 알고리즘의 결과와 비교하였다. 제안하는 알고리즘은 추정된 복원 순서를 바탕으로 구현되어 실험 결과를 통해 복합 열화 문제를 효과적으로 해결할 수 있음을 보인다.

Keywords

BSGHC3_2019_v24n3_430_f0001.png 이미지

그림 1. 제안하는 알고리즘의 흐름도 Fig. 1. Flowchart of the proposed algorithm

BSGHC3_2019_v24n3_430_f0002.png 이미지

그림 2. Inception-v3 네트워크 구조도 Fig. 2. The network diagram of Inception-v3

BSGHC3_2019_v24n3_430_f0003.png 이미지

그림 3. 학습 데이터셋 예시 (a) 안개, (b) 실제 비, (c) 합성된 비, (d) 원본 이미지(DIV2K), (e) 모션 블러, (f) 저 화질, (g) 잡음 Fig. 3. Examples in the training dataset (a) haze, (b) rain real, (c) synthetic rain, (d) original image (DIV2K), (e) motion blurred, (f) low resolution, (g) noise

BSGHC3_2019_v24n3_430_f0004.png 이미지

그림 4. 전이 학습 결과 (a) 정확도, (b) 손실 값 (붉은 색: 학습, 파란 색: 검증) Fig. 4. Transfer learning results (a) accuracy, (b) cross entropy (red: train, blue: validation)

BSGHC3_2019_v24n3_430_f0005.png 이미지

그림 5. 개별 복원 알고리즘 수행 결과 (a) 안개 제거, (b) 비 제거, (c) 블러 제거, (d) 초해상도 복원, (e) 잡음 제거 (왼쪽: 입력 영상, 오른쪽: 복원 결과) Fig. 5. Results of the individual restoration algorithms. (a) dehaze, (b) derain, (c) deblur, (d) SR, (e) denoise (left: input, right: output)

BSGHC3_2019_v24n3_430_f0006.png 이미지

그림 6. 복합 열화 영상 복원 결과 (a) 안개+저화질, (b) 안개+모션 블러, (c) 안개+잡음, (d) 비+저화질, (e) 비+모션 블러, (f) 비+잡음 Fig. 6. Restoration results on multi-degradation image (a) haze+LR, (b) haze+MB, (c) haze+N, (d) rain+LR, (e) rain+MB, (f) rain+N

표 2. 복합 열화 영상 분류 정확도 Table 2. Accuracy on the multi-degradation test dataset

BSGHC3_2019_v24n3_430_t0001.png 이미지

표 3. 복합 열화 영상 복원의 PSNR 평균 및 SSIM 평균 Table 3. PSNR average and SSIM average on the multi-degradation test dataset

BSGHC3_2019_v24n3_430_t0002.png 이미지

표 4. 복합 열화 영상 복원의 CNNIQA 계산 결과 Table 4. CNNIQA evaluation results on the multi-degradation test dataset9

BSGHC3_2019_v24n3_430_t0003.png 이미지

표 1. 단일 열화 영상 분류 정확도 Table 1. Accuracy on the single-degradation test dataset

BSGHC3_2019_v24n3_430_t0004.png 이미지

References

  1. K. Yu, C. Dong, L. Lin, and C. C. Loy, "Crafting a toolchain for image restoration by deep reinforcement learning," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2443-2452, April 2018.
  2. K. He, J. Sun, and X. Tang, "Single image haze removal using dark channel prior," IEEE Trans. On Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp.2341-2353, December 2011 https://doi.org/10.1109/TPAMI.2010.168
  3. B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, "Aod-net: All-in-one dehazing network," Proc. of IEEE International Conference on Computer Vision, pp. 4770-4778, October 2017.
  4. Y. Luo, Y. Xu, and H. Ji, "Removing rain from a single image via discriminative sparse coding," Proc. of IEEE International Conference on Computer Vision, pp. 3397-3405, December 2015.
  5. L. Qiusheng, W. Yan, Z. Xiaohua, and C. Shuzhen, "Single image rain removal using image decomposition and dense network," IEEE/CAA Journal of Automatica Sinica, pp 1-10, March 2019.
  6. G. Huang, Z. Liu, L. Van Der Maaten, and K. Weinberger. "Densely connected convolutional networks," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, July 2017.
  7. S. Cho and S. Lee, "Fast motion deblurring," ACM Trans. on Graphics, vol. 28, no. 5, article 145, December 2009.
  8. X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, "Scale-recurrent network for deep image deblurring," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174-8182, June 2018.
  9. J. Kim, J. K. Lee and K. M. Lee, "Accurate image super-resolution Using very deep convolutional networks," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646-1654, June 2016.
  10. X. Wang, K. Yu, C. Dong, and C. Change, "Recovering realistic texture in image super-resolution by deep spatial feature transform," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 606-615, June 2018.
  11. J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, "Noise2Noise: Learning image restoration without clean data," Proc. of International Conference on Machine Learning, pp. 2971-2980, July 2018.
  12. L. Kang, P. Ye, Y. Li, and D. Doermann, "Convolutional neural networks for no-reference image quality assessment," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1733-1740, June 2014.
  13. C.Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Woina, "Rethinking the inception architecture for computer vision," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, June 2016.
  14. D. Lin, C. Lu, H. Huang, and J. Jia, "RSCM: Region selection and concurrency model for multi-class weather recognition," IEEE Trans. on Image Processing, vol. 26, no. 9, pp. 4154-4167, September 2017. https://doi.org/10.1109/TIP.2017.2695883
  15. X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, "Removing rain from single images via a deep detail network," Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1715-1723, July 2017.
  16. E. Agustsson and R. Timofte, "Ntire 2017 challenge on single image super-resolution: Dataset and study," Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 1122-1131, July 2017.