Journal of Internet of Things and Convergence (사물인터넷융복합논문지)

The Korea Internet of Things Society (한국사물인터넷학회)
권호 표지
DOI QR코드

DOI QR Code

Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh

MediaPipe Face Mesh를 이용한 얼굴 제스처 기반의 사용자 인터페이스의 성능 개선

  • Jinwang Mok (Integrated Candidate of Graduate School of Artificial Intelligence, GIST ) ;
  • Noyoon Kwak (Division of Computer Engineering, Baekseok University)
  • 목진왕 (광주과학기술원 AI대학원) ;
  • 곽노윤 (백석대학교 컴퓨터공학부)
  • Received : 2023.09.29
  • Accepted : 2023.11.23
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

The purpose of this paper is to propose a method to improve the performance of the previous research is characterized by recognizing facial gestures from the 3D coordinates of seven landmarks selected from the MediaPipe Face Mesh model, generating corresponding user events, and executing corresponding commands. The proposed method applied adaptive moving average processing to the cursor positions in the process to stabilize the cursor by alleviating microtremor, and improved performance by blocking temporary opening/closing discrepancies between both eyes when opening and closing both eyes simultaneously. As a result of the usability evaluation of the proposed facial gesture interface, it was confirmed that the average recognition rate of facial gestures was increased to 98.7% compared to 95.8% in the previous research.

본 논문은 MediaPipe Face Mesh 모델을 이용해 일련의 프레임 시퀀스에서 얼굴 제스처를 인식해 해당 사용자 이벤트를 처리하는 얼굴 제스처 기반의 사용자 인터페이스 선행 연구의 성능 개선 방안을 제안함에 그 목적이 있다. 선행 연구는 MediaPipe Face Mesh 모델에서 선택한 7개의 랜드마크의 3차원 좌표들로부터 얼굴 제스처를 인식해 해당 사용자 이벤트를 발생시키고 이에 대응하는 명령을 수행하는 것이 특징이다. 제안된 방법은 그 과정에서 커서 위치들에 적응형 이동 평균 처리를 적용해 미세 떨림을 완화함으로써 커서 안정화를 도모하고, 양안 동시 개폐 시에 양안의 일시적 개폐 불일치를 차단해 그 성능을 개선하였다. 제안된 얼굴 제스처 인터페이스의 사용성 평가 결과, 얼굴 제스처의 평균 인식률이 선행 연구에서 95.8%였던 것에 비해 98.7%로 상향되는 것이 확인되었다.

Keywords

Acknowledgement

본 논문은 2023년도 교육부의 재원으로 한국연구재단의 지원을 받아 수행된 지자체-대학 협력기반 지역혁신 사업의 연구과제(2021RIS-004)로 수행되었음.

References

  1. F. Karray, et al, "Human-Computer Interaction: Overview on State of the Art," International Journal on Smart Sensing and Intelligent Systems, Vol.1, No.1, pp.137-159, 2008.  https://doi.org/10.21307/ijssis-2017-283
  2. T. H. Tsai, C.C. Huang, and K.L. Zhang, "Design of Hand Gesture Recognition System for Human-computer Interaction," Multimedia Tools and Applications, Vol.79, No.9-10, pp.5989-6007, 2020.  https://doi.org/10.1007/s11042-019-08274-w
  3. G. Kim and J. Baek, "Real-Time Hand Gesture Recognition Based on Deep Learning," Journal of Korea Multimedia Society, Vol.22, No.4, pp.424-431, 2019.  https://doi.org/10.9717/KMMS.2019.22.4.424
  4. B. Kumar, R. K. Bedi, and S. K. Gupta, "Facial Gesture Recognition for Emotion Detection: A Review of Methods and Advancements," Handbook of Research on AI-Based Technologies and Applications in the Era of the Metaverse, pp.542-358, 2023. 
  5. Q. Gao, Y. Chen, Z. Ju and Y. Liang, "Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-robot Interaction," IEEE Sensors Journal, pp.17421-17430, 2021. 
  6. H. Kaur and J. Rani, "A Review: Study of Various Techniques of Hand Gesture Recognition," Proceedings of 2016 IEEE 1st International Conference on Power Electronics, Intelligent Control and Energy Systems, pp.1-5, 2016. 
  7. Y. Li, J. Huang, F. Tian, H. Wang, and G. Dai, "Gesture Interaction in Virtual Reality," Virtual Reality and Intelligent Hardware, pp.84-112, 2019. 
  8. C. A. Cruz, N. Tatsuya, M. Ichihara, F. Shibata, and A. Kimura, "Sequential Eyelid Gestures for User Interfaces in VR," Proceedings of 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, Mar. 2023. 
  9. A. Shimada, T. Yamashita and R. Taniguchi, "Hand Gesture Based TV Control System- Towards Both User-Machine-friendly Gesture Applications," Proceedings of The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp.121-126, 2013. 
  10. H. Stern, Y. Edan, M. Gillam, C. Feied, M. Smith, J. Handler, et al., "A Real-Time Hand Gesture Interface for Medical Visualization Applications," Applications of Soft Computing, Vol.36, pp.153-162, Springer, 2006.  https://doi.org/10.1007/978-3-540-36266-1_15
  11. G. Pala, J.B. Jethwani, S.S. Kumbhar, and S. D. Patil, "Machine Learning-based Hand Sign Recognition," Proceedings of 2021 International Conference on Artificial Intelligence and Smart Systems, pp.356-363, 2021. 
  12. M. Iskandar, K. Bingi, B. R. Prusty, M. Omar, and R. Ibrahim, "Artificial Intelligence-based Human Gesture Tracking Control Techniques of Tello EDU Quadrotor Drone," Proceedings of International Conference on Green Energy, Computing and Intelligent Technology, Jul. 2023. 
  13. MANO, https://mano.is.tue.mpg.de (accessed Dec. 10, 2023). 
  14. N. Qian, J. Wang, F. Mueller, F. Bernard, V. Golyanik, C. Theobalt, et al., "HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization," Proceedings of the European Conference on Computer Vision, pp.54-71, 2020. 
  15. S. An, X. Zhang, D. Wei, H. Zhu, J. Yang, K. A. Tsintotas, et al., "Fast Hand: Fast Monocular Hand Pose Estimation on Embedded Systems," Journal of Systems Architecture, Vol.122, 2022. 
  16. Z. Fan, A. Spurr, M. Kocabas, S. Tang, M.J. Black, O. Hilliges, et al., "Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1-10, 2021. 
  17. M. Jang and W. Lee, "Implementation of User Gesture Recognition System for Manipulating a Floating Hologram Character," The Journal of the Institute of Internet, Broadcasting and Communication, Vol.19, No.2, pp.143-149, Feb. 2019.  https://doi.org/10.7236/JIIBC.2019.19.2.143
  18. L. Chen, S.Y. Lin, Y. Xie, Y.Y. Lin, and X. Xie, "MVHM: A Large-Scale multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation," Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.836-845, 2021. 
  19. Google MediaPipe, https://developers.google.com/mediapipe (accessed Dec. 10, 2023). 
  20. K. Heo, B. Song, and J. Hong, "Hierarchical Hand Pose Model for Hand Expression Recognition," Journal of the Korea Institute of Information and Communication Engineering, Vol.25, No.10, pp.1323-1329, 2021.  https://doi.org/10.6109/JKIICE.2021.25.10.1323
  21. K. Heo, M. Kim, B. Song, and B. Shin, "Hand Expression Recognition for Virtual Blackboard," Journal of the Korea Institute of Information and Communication Engineering, Vol.25, No.12, pp.1770-1776, 2021.  https://doi.org/10.6109/JKIICE.2021.25.12.1770
  22. B. Song, S. Lee, H. Choi and S. Kim, "Design and Implementation of a Stereoscopic Image Control System Based on User Hand Gesture Recognition," Journal of the Korea Institute of Information and Communication Engineering, Vol.26, No.3, pp.396-402, 2022.  https://doi.org/10.6109/JKIICE.2022.26.3.396
  23. R. Song, Y. Hong, and N. Kwak, "User Interface Using Hand Gesture Recognition Based on MediaPipe Hands Model," Journal of Korea Multimedia Society, Vol.26, No.2, pp.101-113, Feb. 2023.  https://doi.org/10.9717/kmms.2023.26.2.103
  24. J. Prameela, K. V. Lakshmi, K. Manju, and M. S. Devi, "Mouse Handling Using Facial Gesture," International Research Journal of Modernization in Engineering Technology and Science, Vol.04, No.5, pp.468-475, May 2022. 
  25. S. Sreeni, M. Sabeel, E. S. Kumar, V. H. Vardhan, and K. Chandrakala, "Mouse Cursor Control Using Facial Movements-An HCI Application," International Journal of Techno-Engineering, pp.270-274, Vol.15, No.2, Apr. 2023. 
  26. Z. Sharifisoraki, M. Amini, and S. Rajan, "A Novel Face Recognition Using Specific Values from Deep Neural Network-based Landmarks," Proceedings of 2023 IEEE International Conference on Consumer Electronics, Jan. 2023. 
  27. S. Thino, "Developing a Program to Detect Face Direction and the State of Partially Closed Eyes," Thesis of Master's Degree, Naresuan University, Oct. 2023. 
  28. J. Mok and N. Kwak, "Facial Gesture-based User Interface Using MediaPipe Face Mesh," Proceedings of 2023 Summer Annual Conference of The Institute of Electronics and Information Engineers, pp.1407-1411, Jun. 2023.