Journal of the Institute of Convergence Signal Processing (융합신호처리학회논문지)

The Korea Institute of Convergence Signal Processing (한국융합신호처리학회)
권호 표지
DOI QR코드

DOI QR Code

CG/VR Image Super-Resolution Using Balanced Attention Mechanism

Balanced Attention Mechanism을 활용한 CG/VR 영상의 초해상화

  • Kim, Sowon (Department of Electronic Engineering, Pukyong National University) ;
  • Park, Hanhoon (Department of Electronic Engineering, Pukyong National University)
  • Received : 2021.11.30
  • Accepted : 2021.12.28
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Attention mechanisms have been used in deep learning-based computer vision systems, including single image super-resolution (SISR) networks. However, existing SISR networks with attention mechanism focused on real image super-resolution, so it is hard to know whether they are available for CG or VR images. In this paper, we attempt to apply a recent attention module, called balanced attention mechanism (BAM) module, to 12 state-of-the-art SISR networks, and then check whether the BAM module can achieve performance improvement in CG or VR image super-resolution. In our experiments, it has been confirmed that the performance improvement in CG or VR image super-resolution is limited and depends on data characteristics, size, and network type.

어텐션(Attention) 메커니즘은 딥러닝 기술을 활용한 다양한 컴퓨터 비전 시스템에서 활용되고 있으며, 초해상화(Super-resolution)를 위한 딥러닝 모델에도 어텐션 메커니즘을 적용하고 있다. 하지만 어텐션 메커니즘이 적용된 대부분의 초해상화 기법들은 Real 영상의 초해상화에만 초점을 맞추어서 연구되어, 어텐션 메커니즘을 적용한 초해상화가 CG나 VR 영상 초해상화에도 유효한지는 알기 어렵다. 본 논문에서는 최근에 제안된 어텐션 메커니즘 모듈인 BAM(Balanced Attention Mechanism) 모듈을 12개의 초해상화 딥러닝 모델에 적용한 후, CG나 VR 영상에서도 성능 향상 효과를 보이는지 확인하는 실험을 진행하였다. 실험 결과, BAM 모듈은 제한적으로 CG나 VR 영상의 초해상화 성능 향상에 기여하였으며, 데이터 특징과 크기, 그리고 네트워크 종류에 따라 성능 향상도가 달라진다는 것을 확인할 수 있었다.

Keywords

Acknowledgement

본 연구는 산업통상자원부와 한국산업기술진흥원의 "지역혁신클러스터육성사업(R&D, P0004797)"으로 수행된 연구결과입니다.

References

  1. C. Dong, C. Loy, K. He, and X Tang, "Image super-resolution using deep convolutional networks," PAMI, vol. 38, no. 2, pp. 295-307, 2015.
  2. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, "Image superresolution using very deep residual channel attention networks," ECCV, pp. 286-301, 2018.
  3. Z. Hui, X. Gao, Y. Yang, and X. Wang, "Lightweight image superresolution with information multi-distillation network," ACM MM, pp. 2024-2032, 2019.
  4. H. Zhao, X. Kong, J. He, Y. Qiao, and C. Dong, "Efficient image superresolution using pixel attention," arXiv preprint arXiv:2010.01073, 2020.
  5. S. Anwar and N. Barnes, "Densely residual Laplacian super-resolution," PAMI, 2020.
  6. J. Park, S. Woo, J.Y. Lee, and I.S. Kweon. "BAM: bottleneck attention module," arXiv preprint arXiv:1807.06514, 2018.
  7. F. Wang, H. Hu, and C. Shen, "BAM: a lightweight and efficient balanced attention mechanism for single image super resolution," arXiv preprint arXiv:2104.07566, 2021.
  8. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, "Enhanced deep residual networks for single image super-resolution," CVPRW, pp. 136-144, 2017.
  9. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, "Photo-realistic single image super-resolution using a generative adversarial network," arXiv preprint arXiv:1609.04802, 2016.
  10. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," CVPR, 2016.
  11. N. Ahn, B. Kang, and K. A. Sohn, "Fast, accurate, and lightweight super-resolution with cascading residual network," ECCV, pp. 252-268, 2018.
  12. J. Qin, Y. Huang, and W. Wen, "Multi-scale feature fusion residual network for single image super-resolution," Neurocomputing, vol. 379, pp. 334-342, 2020. https://doi.org/10.1016/j.neucom.2019.10.076
  13. B. Li, B. Wang, J. Liu, Z. Qi, and Y. Shi, "s-LWSR: super lightweight super-resolution network," IEEE Transactions on Image Processing, vol. 29, pp. 8368-8380, 2020. https://doi.org/10.1109/tip.2020.3014953
  14. O. Ronneberger, P. Fischer, and T. Brox, "U-net: convolutional networksfor biomedical image segmentation," International Conference on Medical Image Computing and Computer-assisted Intervention, 2015.
  15. X. He, Z. Mo, P. Wang, Y. Liu, M. Yang, and J. Cheng, "ODE-inspired network design for single image super-resolution," CVPR, pp. 1732-1741, 2020.
  16. C. Wang, Z. Li, and J. Shi, "Lightweight image super-resolution with adaptive weighted learning network," arXiv preprint arXiv:1904.02358, 2019.
  17. A. Allport, "Visual attention," Foundations of Cognitive Science, The MIT Press, pp. 631-682, 1989.
  18. D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics," ICCV, vol. 2, pp. 416-423, 2001.
  19. Reference database, http://level-design.org/referencedb/
  20. M. Bevilacqua, A. Roumy, C. Guillemot, and A. Morel, "Low complexity single image super-resolution based on nonnegative neighbor embedding," BMVC, 2012.
  21. R. Zeyde, M. Elad, and M. Protter, "On single image scale-up using sparse-representations," International Conference on Curves and Surfaces, pp. 711-730, 2010.
  22. S. Kim and H. Park, "Dependency on training data in CG image super-resolution," Summer Conference of Korea Institute of Convergence Signal Processing, pp. 57-59, 2021.
  23. BAM code, https://github.com/dandingbudanding/BAM