• Journal of Internet of Things and Convergence
• /
• v.10 no.5
• /
• pp.1-11
• /
• 2024

This paper describes a hybrid gesture interface based on MediaPipe that recognizes facial gestures and hand gestures using the MediaPipe Hands model and MediaPipe Face Mesh model, and then combines them. First, the presence of hands and faces is determined by individually detecting 3D hand landmarks of MediaPipe Hands model and 3D face landmarks of MediaPipe Face Mesh model from camera input frames, and then the face cursor position and face gestures, as well as the hand cursor position and hand gestures are recognized. Then, these are mixed in a user-friendly way to implement a user interface based on hand-face hybrid gestures. The proposed hand-face hybrid gesture interface based on MediaPipe has the advantage that the gesture mode is set to either the hand or the face, but the interface can be controlled freely using the face and hands without additional gestures for mode switching. In addition, the practicality and usefulness of the proposed hand-face hybrid gesture interface were confirmed through software operation experiments in Windows environment.