Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Adversarial Framework for Joint Light Field Super-resolution and Deblurring

라이트필드 초해상도와 블러 제거의 동시 수행을 위한 적대적 신경망 모델

  • Lumentut, Jonathan Samuel (Inha University, Department of Information and Communication Engineering) ;
  • Baek, Hyungsun (Inha University, Department of Information and Communication Engineering) ;
  • Park, In Kyu (Inha University, Department of Information and Communication Engineering)
  • Received : 2020.07.13
  • Accepted : 2020.08.28
  • Published : 2020.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Restoring a low resolution and motion blurred light field has become essential due to the growing works on parallax-based image processing. These tasks are known as light-field enhancement process. Unfortunately, only a few state-of-the-art methods are introduced to solve the multiple problems jointly. In this work, we design a framework that jointly solves light field spatial super-resolution and motion deblurring tasks. Particularly, we generate a straight-forward neural network that is trained under low-resolution and 6-degree-of-freedom (6-DOF) motion-blurred light field dataset. Furthermore, we propose the strategy of local region optimization on the adversarial network to boost the performance. We evaluate our method through both quantitative and qualitative measurements and exhibit superior performance compared to the state-of-the-art methods.

시차 기반 영상처리에 대한 연구들이 증가함에 따라 저해상도 및 모션 블러된 라이트필드 영상을 복원하는 연구는 필수적이 되었다. 이러한 기법들은 라이트필드 영상 향상 과정으로 알려져 있으나 두 개 이상의 문제를 동시에 해결하는 기존의 연구는 거의 존재하지 않는다. 본 논문에서는 라이트필드 공간 영역 초해상도 복원과 모션 블러 제거를 동시 수행하는 프레임워크를 제안한다. 특히, 저해상도 및 6-DOF 모션 블러된 라이트필드 데이터셋으로 훈련하는 간단한 네트워크를 생성한다. 또한 성능을 향상하기 위해 생성적 적대 신경망의 지역 영역 최적화 기법을 제안하였다. 제안한 프레임워크는 정량적, 정성적 측정을 통해 평가하고 기존의 state-of-the-art 기법들과 비교하여 우수한 성능을 나타냄을 보인다.

Keywords

References

  1. Y. Bok, H. G. Jeon, and I. S. Kweon, "Geometric calibration of micro-lense-based light field cameras using line features," IEEE Trans. on Pattern Analysis and Machine Intelligence, 39(2):287-300, February 2017. https://doi.org/10.1109/TPAMI.2016.2541145
  2. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," In Proc. of Advances in Neural Information Processing Systems, pages 2672-2680, December 2014.
  3. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of Wasserstein GANs," In Proc. of Advances in Neural Information Processing Systems, pages 5767-5777, December 2017.
  4. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, June 2016.
  5. T. H. Kim, K. M. Lee, B. Scholkopf, and M. Hirsch, "Online video deblurring via dynamic temporal blending network," In Proc. of IEEE International Conference on Computer Vision, pages 4058-4067, 2017.
  6. S. Iizuka, E. Simo-Serra, and H. Ishikawa, "Globally and locally consistent image completion," ACM Transactions on Graphics, 36(4):107, Article No. 107, July 2017.
  7. P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1125-1134, July 2017.
  8. H. G. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y. W. Tai, and I. S. Kweon, "Accurate depth map estimation from a lenslet light field camera," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1547-1555, June 2015.
  9. N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, "Learning-based view synthesis for light field cameras," ACM Transactions on Graphics, 35(6), Article No. 193, November 2016.
  10. J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1646-1654, June 2016.
  11. D. Krishnan, T. Tay, and R. Fergus, "Blind deconvolution using a normalized sparsity measure," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 233-240, June 2011.
  12. O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, "DeblurGAN: Blind motion deblurring using conditional adversarial networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8183-8192, June 2018.
  13. W. S. Lai, J. B. Huang, Z. Hu, N. Ahuja, and M. H. Yang, "A comparative study for single image blind deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1701-1709, June 2016.
  14. D. Lee, H. Park, I. K. Park, and K. M. Lee, "Joint blind motion deblurring and depth estimation of light field," In Proc. of European Conference on Computer Vision, pp. 300-316, September 2018.
  15. M. Levoy and P. Hanrahan, "Light field rendering," In Proc. of SIGGRAPH, pages 31-42, August 1996.
  16. J. S. Lumentut, T. H. Kim, R. Ramamoorthi, and I. K. Park, "Fast and full-resolution light field deblurring using a deep neural network," IEEE Signal of Processing Letters, 26(12), pages 1788-1792, December 2019. https://doi.org/10.1109/LSP.2019.2947379
  17. J.S. Lumentut, Williem, and I. K. Park, "6-DOF Motion Blur Synthesis and Performance Evaluation of Light Field Deblurring," Multimedia Tools and Application, 78(23), pages 33723-33746, December 2019. https://doi.org/10.1007/s11042-019-08030-0
  18. M. R. Mahesh Mohan and A. N. Rajagopalan, "Divide and conquer for full-resolution light field deblurring," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6421-6429, June 2018.
  19. S. Nah, T. H. Kim, and K. M. Lee, "Deep multi-scale convolutional neural network for dynamic scene deblurring," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 3883-3891, July 2017.
  20. R. Ng, "Fourier slice photography," ACM Trans. on Graphics, 24(3):735-744, July 2005. https://doi.org/10.1145/1073204.1073256
  21. J. Pan, Z. Hu, Z. Su, and M. H. Yang, "Deblurring text images via L0-regularized intensity and gradient prior," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2908, June 2014.
  22. W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y. Wang, "UnrealCV: Virtual worlds for computer vision." In Proc. of ACM International Conference on Multimedia, pages 1221-1224, October 2017.
  23. Z. Shen, W. S. Lai, T. Xu, J. Kautz, and M. H. Yang, "Deep semantic face deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 8260-8269, June 2018.
  24. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," In Proc. of International Conference on Learning Representations, April 2014.
  25. P. P. Srinivasan, R. Ng, and R. Ramamoorthi, "Light field blind motion deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 3958-3966, July 2017.
  26. P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, "Pushing the boundaries of view extrapolation with multiplane images," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 175-184, June 2019.
  27. P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng, "Learning to synthesize a 4D RGBD light field from a single image," In Proc. of IEEE International Conference on Computer Vision, pages 2262-2270, October 2017.
  28. S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang, "Deep video deblurring for hand-held cameras," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 237-246, July 2017.
  29. Y. W. Tai, P. Tan, and M. S. Brown, "Richardson-Lucy deblurring for scenes under a projective motion path," IEEE Trans. on Pattern Analysis and Machine Intelligence, 33(8):1603-1618, August 2011. https://doi.org/10.1109/TPAMI.2010.222
  30. M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, "Depth from combining defocus and correspondence using light-field cameras," In Proc. of the IEEE International Conference on Computer Vision, pages 673-680, December 2013.
  31. M. W. Tao, P. P Srinivasan, J. Malik, S. Rusinkiewicz, and R. Ramamoorthi, "Depth from shading, defocus, and correspondence using light-field angular coherence," In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1940-1948, June 2015.
  32. D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv preprint arXiv:1607.08022, 2016.
  33. T. C Wang, A. A. Efros, and Ravi Ramamoorthi, "Depth estimation with occlusion modeling using light-field cameras," IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(11):2170-2181, November 2016. https://doi.org/10.1109/TPAMI.2016.2515615
  34. Williem, I. K. Park, and K. M. Lee, "Robust light field depth estimation using occlusion-noise aware data costs," IEEE Trans. on Pattern Analysis and Machine Intelligence, 40(10):2484-2497, October 2018. https://doi.org/10.1109/TPAMI.2017.2746858
  35. Y. Yoon, H. G. Jeon, D. Yoo, J. Y. Lee, and I. S. Kweon, "Learning a deep convolutional network for light-field image super-resolution," In Proc. of IEEE International Conference on Computer Vision Workshop, pages 57-65, June 2015.
  36. S. Zhang, Y. Lin, and H. Sheng, "Residual networks for light field image super-resolution," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 11046-11055, June 2019.
  37. T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, "Stereo magnification: Learning view synthesis using multiplane images," ACM Trans. on Graphics, 37(4), Article No. 65, July 2018.