Journal of Korean Society of Industrial and Systems Engineering (산업경영시스템학회지)

Society of Korea Industrial and System Engineering (한국산업경영시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Dataset Evaluation Criteria for Learning Deepfake Video

딥페이크 영상 학습을 위한 데이터셋 평가기준 개발

  • Kim, Rayng-Hyung (Department. of Industiral & Management Enginering, Hanbat National University) ;
  • Kim, Tae-Gu (Department. of Industiral & Management Enginering, Hanbat National University)
  • Received : 2021.11.24
  • Accepted : 2021.12.24
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

As Deepfakes phenomenon is spreading worldwide mainly through videos in web platforms and it is urgent to address the issue on time. More recently, researchers have extensively discussed deepfake video datasets. However, it has been pointed out that the existing Deepfake datasets do not properly reflect the potential threat and realism due to various limitations. Although there is a need for research that establishes an agreed-upon concept for high-quality datasets or suggests evaluation criterion, there are still handful studies which examined it to-date. Therefore, this study focused on the development of the evaluation criterion for the Deepfake video dataset. In this study, the fitness of the Deepfake dataset was presented and evaluation criterions were derived through the review of previous studies. AHP structuralization and analysis were performed to advance the evaluation criterion. The results showed that Facial Expression, Validation, and Data Characteristics are important determinants of data quality. This is interpreted as a result that reflects the importance of minimizing defects and presenting results based on scientific methods when evaluating quality. This study has implications in that it suggests the fitness and evaluation criterion of the Deepfake dataset. Since the evaluation criterion presented in this study was derived based on the items considered in previous studies, it is thought that all evaluation criterions will be effective for quality improvement. It is also expected to be used as criteria for selecting an appropriate deefake dataset or as a reference for designing a Deepfake data benchmark. This study could not apply the presented evaluation criterion to existing Deepfake datasets. In future research, the proposed evaluation criterion will be applied to existing datasets to evaluate the strengths and weaknesses of each dataset, and to consider what implications there will be when used in Deepfake research.

Keywords

Acknowledgement

This research was supported by the research fund of Hanbat National University in 2021.

References

  1. Dolhansky, B., Howes, R., Pflaum, B., Baram, N., and Ferrer, C. C., The Deepfake detection challenge (dfdc) preview dataset, arXiv preprint arXiv:1910.08854., 2019.
  2. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C.C., The Deepfake detection challenge (dfdc) dataset, arXiv preprint arXiv:2006.07397, 2020.
  3. Dordevic, M., Milivojevic, M., and Gavrovska, A., Deepfake video analysis using SIFT features, In 2019 27th Telecommunications Forum (TELFOR), IEEE, 2019, November, pp. 1-4.
  4. Dufour, N. and Gully, A., Contributing data to deep-fake detection research, 2019. URL https://ai.googleblog.com/2019/09/contributing-data-to-Deepfake-detection.html.
  5. Feng, K., Wu, J., and Tian, M., A Detect method for Deepfake video based on full face recognition, In 2020 IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), IEEE, 2020, November, Vol. 1, pp. 1121-1125.
  6. Guarnera, L., Giudice, O., and Battiato, S., Fighting Deepfake by exposing the convolutional traces on images, IEEE Access, 2020, Vol. 8, pp. 165085-165098. https://doi.org/10.1109/access.2020.3023037
  7. Jiang, L., Li, R., Wu, W., Qian, C., and Loy, C. C., Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2889-2898.
  8. Karras, T., Aila, T., Laine, S., and Lehtinen, J., Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  9. Kim, H. S. A study on the data quality management evaluation model. Journal of the Korea Convergence Society, 2020, Vol. 11, No. 7, pp. 217-222. https://doi.org/10.15207/JKCS.2020.11.7.217
  10. Kim, T.G., Kim, D.H., Lee, D.J., Kim, K.T., and Moon, S.D.S., Development of Checklist to Promote Commercialization of R&D for Railway Transportation using AHP method, Journal of the Korea Management Engineers Society, 2017, Vol. 22, No. 1, pp. 77-87. https://doi.org/10.35373/KMES.22.1.5
  11. KOREA Data Agency, Guideline for data quality management(Ver 2.1), 2006.
  12. Korshunov, P., and Marcel, S., Vulnerability assessment and detection of Deepfake videos, In 2019 International Conference on Biometrics (ICB), IEEE, 2019, June, pp. 1-6.
  13. Kwon, P., You, J., Nam, G., Park, S., and Chae, G., KoDF: A Large-scale Korean Deepfake Detection Dataset, arXiv preprint arXiv:2103.10094, 2021.
  14. Le, T. N., Nguyen, H. H., Yamagishi, J., and Echizen, I. OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10117-10127.
  15. Li, Y., and Lyu, S., Exposing Deepfake videos by detecting face warping artifacts, ArXiv Preprint ArXiv:1811.00656, 2018.
  16. Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, S., Celeb-df: A large-scale challenging dataset for Deepfake forensics, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3207-3216.
  17. Mahfoudi, G., Tajini, B., Retraint, F., Morain-Nicolier, F., Dugelay, J.L., and Marc, P.I.C., DEFACTO: Image and face manipulation dataset, In 2019 27th European Signal Processing Conference (EUSIPCO), IEEE, 2019, September, pp. 1-5.
  18. Matern, F., Riess, C., and Stamminger, M., Exploiting visual artifacts to expose Deepfakes and face manipulations, In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), IEEE, 2019, January, pp. 83-92.
  19. National Information Society Agency, Guideline for artificial intelligence learning data quality management, 2020.
  20. Park, G. and Kim, C.J., Quality Characteristics of Public Open Data, Journal of Digital Convergence, 2015, Vol. 13, No. 10, pp. 135-146. https://doi.org/10.14400/JDC.2015.13.10.135
  21. Ramadhani, K.N. and Munir, R., A Comparative Study of Deepfake Video Detection Method, In 2020 3rd International Conference on Information and Communications Technology (ICOIACT), IEEE, 2020, November, pp. 394-399.
  22. Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Niessner, M., Faceforensics++: Learning to detect manipulated facial images, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1-11.
  23. Saaty, T.L., Axiomatic foundation of the analytic hierarchy process, Management Science, 1986, Vol. 32, No. 7, pp. 841- 855. https://doi.org/10.1287/mnsc.32.7.841
  24. Tayseer, M., Mohammad, J., Ababneh, M., Al-Zoube, A., and Elhassan, A., Digital Forensics and Analysis of Deepfake Videos, In 11th International Conference on Information and Communication Systems (ICICS), 2020.
  25. Thies, J., Zollhofer, M., and Niessner, M., Deferred neural rendering, Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 2019, Vol. 38, No. 4, pp. 1-12.
  26. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., and Niessner, M., Face2face: Real-time face capture and reenactment of rgb videos, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2387-2395.
  27. Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., and Ortega-Garcia, J., Deepfakes and beyond: A survey of face manipulation and fake detection, Information Fusion, 2020, Vol. 64, pp. 131-148. https://doi.org/10.1016/j.inffus.2020.06.014
  28. Verdoliva, L., Media Forensics and Deepfakes: An Overview, IEEE Journal of Selected Topics in Signal Processing, 2020, Vol. 14, No. 5, pp. 910-932. https://doi.org/10.1109/jstsp.2020.3002101
  29. Wang, R., Juefei-Xu, F., Ma, L., Xie, X., Huang, Y., Wang, J., and Liu, Y., Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces, ArXiv Preprint ArXiv:1909.06122, 2019.
  30. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B., High-resolution image synthesis and semantic manipulation with conditional gans, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8798-8807.
  31. Yang, J.Y. and Lee, S.R., A Study on the Priority-Gap Measurement of Performance Factors Before and After Introduction of Electronic Price Information System in Retail Stores using IT-BSC and AHP, Information Systems Review, 2020, Vol. 22, No. 2, pp. 53-76. https://doi.org/10.14329/isr.2020.22.2.053
  32. Youm, G.Y. and Kim, M.C., SAR image target recognition research trend using deep learning, The Journal of The Korean Institute of Communication Sciences, 2017, Vol. 34, No. 7, pp. 31-39.
  33. Yu, P., Xia, Z., Fei, J., and Lu, Y., A Survey on Deepfake Video Detection, IET Biometrics, 2016, October.
  34. Yun, C.H., Shin, H.Y., Choo, S.Y., and Kim, J.I., An evaluation study on artificial intelligence data validation methods and open-source frameworks, Journal of Korea Multimedia Society, 2021, Vol. 24, No. 10, pp. 1403-1413. https://doi.org/10.9717/KMMS.2021.24.10.1403
  35. Zi, B., Chang, M., Chen, J., Ma, X., and Jiang, Y.G., WildDeepfake: A challenging real-world dataset for Deepfake detection, In Proceedings of the 28th ACM International Conference on Multimedia, 2020, October, pp. 2382-2390.
  36. UCI Machine Learning Repository, https://archive.ics.uci.edu/ml/datasets.php.
  37. Kaggle, https://www.kaggle.com/.