Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

A New Image Processing Scheme For Face Swapping Using CycleGAN

순환 적대적 생성 신경망을 이용한 안면 교체를 위한 새로운 이미지 처리 기법

  • Ban, Tae-Won (Department of Intelligent Communication Engineering, Gyeongsang National University)
  • Received : 2022.08.05
  • Accepted : 2022.08.22
  • Published : 2022.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

With the recent rapid development of mobile terminals and personal computers and the advent of neural network technology, real-time face swapping using images has become possible. In particular, the cycle generative adversarial network made it possible to replace faces using uncorrelated image data. In this paper, we propose an input data processing scheme that can improve the quality of face swapping with less training data and time. The proposed scheme can improve the image quality while preserving facial structure and expression information by combining facial landmarks extracted through a pre-trained neural network with major information that affects the structure and expression of the face. Using the blind/referenceless image spatial quality evaluator (BRISQUE) score, which is one of the AI-based non-reference quality metrics, we quantitatively analyze the performance of the proposed scheme and compare it to the conventional schemes. According to the numerical results, the proposed scheme obtained BRISQUE scores improved by about 4.6% to 14.6%, compared to the conventional schemes.

최근 모바일 단말기 및 개인형 컴퓨터의 비약적인 발전과 신경망 기술의 등장으로 영상을 활용한 실시간 안면 교체가 가능해졌다. 특히, 순환 적대적 생성 신경망은 상호 연관성이 없는 이미지 데이터를 활용한 안면 교체가 가능하게 만들었다. 본 논문에서는 적은 학습 데이터와 시간으로 안면 교체의 품질을 높일 수 있는 입력 데이터 처리 기법을 제안한다. 제안 방식은 사전에 학습된 신경망을 통해서 추출된 안면의 특이점 정보와 안면의 구조와 표정에 영향을 미치는 주요 이미지 정보를 결합함으로써 안면 표정과 구조를 보존하면서 이미지 품질을 향상시킬 수 있다. 인공지능 기반의 무참조 품질 메트릭 중의 하나인 blind/referenceless image spatial quality evaluator (BRISQUE) 점수를 활용하여 제안 방식의 성능을 정량적으로 분석하고 기존 방식과 비교한다. 성능 분석 결과에 따르면 제안 방식은 기존 방식 대비 약 4.6%~14.6% 개선된 BRISQUE 점수를 나타내었다.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government(Ministry of Education) (No. 2020R1I1A3061195).

References

  1. Y. Zhang, L. Zheng, and V. L. L. Thing, "Automated Face Swapping and Its Detection," in Proceeding of IEEE 2nd International Conference on Signal and Image, Singapore, pp. 15-19, 2017.
  2. Q. Zhang, H. Zheng, T. Yan, and J. Li, "3D Large-Pose Face Alignment Method Based on the Truncated Alexnet Cascade Network," Advances in Condensed Matter Physics, vol. 2020, Article ID 6675014, Dec. 2020.
  3. D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, and S. K. Nayar, "Face Swapping: Automatically Replacing Faces in Photographs," ACM Transactions on Graphics (SIGGRAPH), vol. 27, no. 3, pp. 1-8, Aug. 2008.
  4. K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister, "Video Face Replacement," ACM Transactions on Graphics (SIGGRAPH), vol. 30, no. 6, pp. 1-10, Dec. 2011.
  5. S. Suwajanakorn, S. M. Seitz, and I. KemelmacherShlizerman, "What Makes Tom Hanks Look Like Tom Hanks," in Proceeding of IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 3952-3960, 2015.
  6. I. Korshunova, W. Shi, J. Dambre, and L. Theis, "Fast Face-Swap Using Convolutional Neural Networks," in Proceeding of IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 3697-3705, 2017.
  7. A. Datta, O. K. Yadav, Y. Singh, S. S, K. M, and S. E, "Real-Time Face Swapping System using OpenCV," in Proceeding of the 3rd International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, pp. 1081-1086, 2021.
  8. github, 2020. Github [Internet]. Available: https://github.com/JunHyeok96/DeepFake.
  9. X. Jin, Y. Qi, and S. Wu, "CycleGAN Face-off," arXiv. [Internet]. Available: https://arxiv.org/abs/1712.03451.
  10. J. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks," in Proceeding of 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 2242-2251, 2017.
  11. github, 2021. Github [Internet]. Available: https://github.com/nicolalandro/cyclegan-pretrained
  12. dlib C++ Library[Internet]. Available: http://dlib.net/.
  13. A. Mittal, A. K. Moorthy, and A. C. Bovik, "No-Reference Image Quality Assessment in the Spatial Domain," IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695-4708, Dec. 2012. https://doi.org/10.1109/TIP.2012.2214050
  14. github, 2022. Github [Internet]. Available: https://github.com/photosynthesis-team/piq.
  15. [Internet]. Available: https://youtu.be/DXdzTS6ZHhw.