Journal of the Institute of Convergence Signal Processing (융합신호처리학회논문지)

The Korea Institute of Convergence Signal Processing (한국융합신호처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Comparative Study of Knowledge Distillation Methods in Lightening a Super-Resolution Model

초해상화 모델 경량화를 위한 지식 증류 방법의 비교 연구

  • Yeojin Lee (Division of Electronics and Communications Engineering, Pukyong National University) ;
  • Hanhoon Park (Division of Electronics and Communications Engineering, Pukyong National University)
  • 이여진 (부경대학교 전자정보통신공학부) ;
  • 박한훈 (부경대학교 전자정보통신공학부)
  • Received : 2022.11.28
  • Accepted : 2023.01.10
  • Published : 2023.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

Knowledge distillation (KD) is a model lightening technology that transfers the knowledge of deep models to light models. Most KD methods have been developed for classification models, and there have been few KD studies in the field of super-resolution (SR). In this paper, various KD methods are applied to an SR model and their performance is compared. Specifically, we modified the loss function to apply each KD method to the SR model and conducted an experiment to learn a student model that was about 27 times lighter than the teacher model and to double the image resolution. Through the experiment, it was confirmed that some KD methods were not valid when applied to SR models, and that the performance was the highest when the relational KD and the traditional KD methods were combined.

지식 증류는 깊은 모델의 지식을 가벼운 모델로 전달하는 모델 경량화 기술이다. 대부분의 지식 증류 방법들은 분류 모델을 위해 개발되었으며, 초해상화를 위한 지식 증류 연구는 거의 없었다. 본 논문에서는 다양한 지식 증류 방법들을 초해상화 모델에 적용하고 성능을 비교한다. 구체적으로, 초해상화 모델에 각 지식 증류 방법을 적용하기 위해 손실 함수를 수정하고, 각 지식 증류 방법을 사용하여 교사 모델을 약 27배 경량화한 학생 모델을 학습하여 2배 초해상화하는 실험을 진행하였다. 실험을 통해, 일부 지식 증류 방법은 초해상화 모델에 적용할 경우 유효하지 않음을 알 수 있었으며, 관계 기반 지식 증류 방법과 전통적인 지식 증류 방법을 결합했을 때 성능이 가장 높은 것을 확인하였다.

Keywords

Acknowledgement

본 연구는 산업통상자원부와 한국산업기술진흥원의 "지역혁신클러스터육성사업(R&D, P0004797)"으로 수행된 연구결과 입니다.

References

  1. G. Hinton, et al., "Distilling the knowledge in a neural network," Proc. of NIPS, 2014.
  2. C. Ledig, et al., "Photo-realistic single image super-resolution using a generative adversarial network," Proc. of CVPR, pp. 105-114, 2017.
  3. B. Lim, et al., "Enhanced deep residual networks for single image super-resolution," Proc. of CVPRW, pp. 136-144, 2017.
  4. A. Romero, et al., "FitNets: hints for thin deep nets," Proc. of ICLR, 2015.
  5. N. Komodakis and S. Zagoruyko, "Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer," Proc. of ICLR, 2017.
  6. Y. Zhang, et al.,"Deep mutual learning," Proc. of CVPR, pp. 4320-4328, 2018.
  7. L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, "Be your own teacher: improve the performance of convolutional neural networks via self distillation," Proc. of ICCV, pp. 3713-3722, 2019.
  8. Z. He, et al., "Fakd: feature-affinity based knowledge distillation for efficient image super-resolution," Proc. of ICIP, pp. 518-522, 2020.
  9. H. Moon, et al., "Compression of super-resolution model using contrastive learning," Proc. of KIBME Summer Conference, pp. 557-559, 2022.
  10. W. Park, et al., "Relational knowledge distillation," Proc. of CVPR, pp. 3967-3976, 2019.
  11. P. Chen, et al., "Distilling knowledge via knowledge review," Proc. of CVPR, 2021.
  12. E. Agustsson and R. Timofte, "Ntire 2017 challenge on single image super-resolution: dataset and study," Proc. of CVPRW, pp. 126-135, 2017.
  13. M. Bevilacqua, et al., "Low complexity single image super-resolution based on non negative neighbor embedding," Proc. of BMVC, 2012.
  14. R. Zeyde, et al., "On single image scale-up using sparse-representations," Proc. of Int. Conf. on Curves and Surfaces, pp. 711-730, 2010.
  15. D. Martin, et al., "A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics," Proc. of ICCV, vol. 2, pp. 416-423, 2001.
  16. J.-B. Huang, et al., "Single image super-resolution from transformed self-exemplars," Proc. of CVPR, pp. 5197-5206, 2015.