Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Comparative analysis of the deep-learning-based super-resolution methods for generating high-resolution texture maps

고해상도 텍스처 맵 생성을 위한 딥러닝 기반 초해상도 기법들의 비교 분석 연구

  • Received : 2023.04.17
  • Accepted : 2023.08.31
  • Published : 2023.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

As display resolution increases, many apps also tend to include high-resolution texture maps. Recent advancements in deep-learning-based image super-resolution techniques make it possible to automate high-resolution texture generation. However, there is still a lack of comprehensive analysis of the application of these techniques to texture maps. In this paper, we selected three recent super-resolution techniques, namely BSRGAN, Real-ESRGAN, and SwinIR (classical and real-world image SR), and applied them to upscale texture maps. We then conducted a quantitative and qualitative analysis of the experimental results. The findings revealed various artifacts after upscaling, which indicates that there are still limitations in directly applying super-resolution techniques to texture-map upscaling.

디스플레이의 해상도의 증가에 따라 고해상도 텍스처 맵을 내장한 앱들도 함께 증가하는 추세에 있다. 최근 딥러닝 기반 이미지 초해상도 기법들의 발전은 이러한 고해상도 텍스처 생성 작업을 자동화할 수 있는 가능성을 만들고 있다. 하지만 이러한 적용 사례에 대해 심층적으로 분석한 연구는 아직 부족한 상태이다. 그래서 본 논문에서는 최신 초해상도 기법들 중 BSRGAN, Real-ESRGAN, SwinIR(classical 및 real-world image SR)을 택하여 텍스처 맵의 업스케일링(upscaling)에 적용한 후 그 결과를 정량적, 정성적으로 비교, 분석하였다. 실험 결과 업스케일링 후 나타나는 다양한 아티팩트(artifact)들을 발견할 수 있었으며, 이를 통해 기존 초해상도 기법들을 텍스처 맵 업스케일링에 곧바로 적용하기에는 일부 미흡한 부분이 존재한다는 점을 확인하였다.

Keywords

Acknowledgement

본 연구는 2023학년도 상명대학교 교내연구비를 지원받아 수행하였음 (2023-A000-0318). 실험 데이터는 Kodak, Crytek, UNC GAMMA Lab, Spiral Graphics, Ignacio Castano가 공개한 이미지 및 장면을 포함함.

References

  1. J. Papadopoulos. (2023) New Cyberpunk 2077 HD texture packs improve the textures of NPCs faces, hair & beards. [Online]. Available: https://www.dsogaming.com/mods/new-cyberpunk-2077-hd-texture-packs-improve-the-textures-of-npcs-faces-hair-beards/ 
  2. GPUnity. (2023) Arkham city redux. [Online]. Available: https://www.nexusmods.com/batmanarkhamcity/mods/407/ 
  3. L. Williams, "Pyramidal parametrics," in Proceedings of the 10th annual conference on Computer graphics and interactive techniques (SIGGRAPH), 1983, pp. 1-11. 
  4. S. Takemura, "Optimize deep super-resolution and denoising for compressed textures," in ACM SIGGRAPH Asia 2018 Posters, 2018. 
  5. J.-H. Nah and H. Kim, "TexSR: Image super-resolution for high-quality texture mapping," in ACM SIGGRAPH Asia 2022 Posters, 2022. 
  6. K. Zhang, J. Liang, L. V. Gool, and R. Timofte, "Designing a practical degradation model for deep blind image super-resolution," in Proceedings of the International Conference on Computer Vision (ICCV), 2021, pp. 4791-4800. 
  7. X. Wang, L. Xie, C. Dong, and Y. Shan, "Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data," in 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021, pp. 1905-1914. 
  8. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, "SwinIR: Image restoration using swin transformer," in 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021, pp. 1833-1844. 
  9. P. Andersson, J. Nilsson, T. Akenine-Moller, M. Oskarsson, K. Astrom, and M. D. Fairchild, "FLIP: A difference evaluator for alternating images," Proceedings of the ACM on Computer Graphics and Interactive Techniques (HPG), vol. 3, no. 2, pp. 1-23, 2020. 
  10. NVIDIA Developer. (2023) Deep learning super sampling (DLSS). [Online]. Available: https://www.nvidia.com/en-us/geforce/technologies/dlss/ 
  11. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," In Proceedings of European Conference on Computer Vision (ECCV) 2014, pp. 184-199, 2014. 
  12. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, "Photo-realistic single image super-resolution using a generative adversarial network," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 105-114. 
  13. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy, "ESRGAN: Enhanced super-resolution generative adversarial networks," in Computer Vision - ECCV 2018 Workshops, 2019, pp. 63-79. 
  14. X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang, "Realworld super-resolution via kernel estimation and noise injection," in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020, pp. 1914-1923. 
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," 2017. 
  16. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 9992-10 002. 
  17. J.-H. Nah, "ASTC block-size determination method based on PSNR values," Journal of the Korea Computer Graphics Society, vol. 28, no. 2, pp. 21-28, 2022.  https://doi.org/10.15701/kcgs.2022.28.2.21
  18. W. Cho, J. Kwon, S. Kwon, and J. Yoo, "A comparative study on OCR using super-resolution for small fonts," The International Journal of Advanced Smart Convergence, vol. 8, no. 3, pp. 95-101, 2019.  https://doi.org/10.7236/IJASC.2019.8.3.95
  19. S.-H. Sung, K.-B. Lee, and S.-H. Park, "Research on Korea text recognition in images using deep learning," Journal of the Korea Convergence Society, vol. 11, no. 6, pp. 1-6, 2020.  https://doi.org/10.15207/JKCS.2020.11.6.001
  20. Y. Li, B. Sixou, and F. Peyrin, "A review of the deep learning methods for medical images super resolution problems," IRBM, vol. 42, no. 2, pp. 120-133, 2021.  https://doi.org/10.1016/j.irbm.2020.08.004
  21. S. Jia, Z. Wang, Q. Li, X. Jia, and M. Xu, "Multiattention generative adversarial network for remote sensing image super-resolution," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022.  https://doi.org/10.1109/TGRS.2022.3180068
  22. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.  https://doi.org/10.1109/TIP.2003.819861
  23. J.-H. Nah, "QuickETC2: How to finish ETC2 compression within 1 ms," in ACM SIGGRAPH 2020 Talks, 2020. 
  24. J.-H. Nah, "QuickETC2: Fast ETC2 texture compression using luma differences," ACM Transactions on Graphics (SIGGRAPH Asia 2020), vol. 39, no. 6, p. 10, nov 2020. 
  25. J. M. P. van Waveren and I. Castano, "Real-time normal map DXT compression," Technical Report, Id Software, 2008. [Online]. Available: https://developer.download.nvidia.com/whitepapers/2008/realtime-normal-map-dxt-compression.pdf 
  26. I. Castano. (2021) Image datasets - a repository of public image data sets. [Online]. Available: https://github.com/castano/image-datasets 
  27. E. Agustsson and R. Timofte, "NTIRE 2017 challenge on single image super-resolution: Dataset and study," in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 1122-1131. 
  28. A. Gianluca. (2023) QualityScaler - image/video AI upscaler app (BSRGAN). [Online]. Available: https://github.com/Djdefrag/QualityScaler/ 
  29. S. Yuk and Y. Cho, "A study on novel steganography communication technique based on thumbnail images in sns messenger environment," Journal of Internet Computing and Services, vol. 22, no. 6, pp. 151-162, 2021.  https://doi.org/10.7472/JKSII.2021.22.6.151
  30. J. Nilsson and T. Akenine-Moller, "Understanding SSIM," ArXiv e-prints, 2020, arXiv:2006.13846. 
  31. NVIDIA Developer. (2023) DLSS - download and get started. [Online]. Available: https://developer.nvidia.com/rtx/dlss/get-started 
  32. J.-H. Nah, B. Choi, and Y. Lim, "Classified texture resizing for mobile devices," in ACM SIGGRAPH 2018 Talks, New York, NY, USA, 2018. [Online]. Available: https://doi.org/10.1145/3214745.3214763 
  33. LG Electronics. (2022) α9 Gen5 AI Processor. [Online]. Available: https://www.lg.com/levanten/tvs/2022/alpha9 
  34. K. Vaidyanathan, M. Salvi, B. Wronski, T. Akenine-Moller, P. Ebelin, and A. Lefohn, "Random-access neural compression of material textures," ACM Transaction on Graphics, vol. 42, no. 4, jul 2023. [Online]. Available: https://doi.org/10.1145/3592407 
  35. J.-H. Nah, "Addition of an adaptive block-size determination feature to astcenc, the reference ASTC encoder," Software Impacts, vol. 17, 2023.