Face and Hand Tracking Algorithm for Sign Language Recognition

수화 인식을 위한 얼굴과 손 추적 알고리즘

  • 박호식 (관동대학교 전자통신공학과 영상처리연구실) ;
  • 배철수 (관동대학교 전자통신공학과 영상처리연구실)
  • Published : 2006.11.30


In this paper, we develop face and hand tracking for sign language recognition system. The system is divided into two stages; the initial and tracking stages. In initial stage, we use the skin feature to localize face and hands of signer. The ellipse model on CbCr space is constructed and used to detect skin color. After the skin regions have been segmented, face and hand blobs are defined by using size and facial feature with the assumption that the movement of face is less than that of hands in this signing scenario. In tracking stage, the motion estimation is applied only hand blobs, in which first and second derivative are used to compute the position of prediction of hands. We observed that there are errors in the value of tracking position between two consecutive frames in which velocity has changed abruptly. To improve the tracking performance, our proposed algorithm compensates the error of tracking position by using adaptive search area to re-compute the hand blobs. The experimental results indicate that our proposed method is able to decrease the prediction error up to 96.87% with negligible increase in computational complexity of up to 4%.


  1. V. I. Pavlovic, R. Sharma, T. S. Huang, 'Visual interpretation of hand gestures for human-computer interaction: a review' IEEE transactions on pattern analysis and machine intelligence, v.19 no.7, pp.677-695, 1997 https://doi.org/10.1109/34.598226
  2. D. M. Gavrila, 'The Visual Analysis of Human Movement: A Survey' CVIU, v.73 no.1, pp.82-98, 1999 https://doi.org/10.1006/cviu.1998.0716
  3. A. Shamaie and A. Sutherland, 'A dynamic model for real-time tracking of hands in bimanual movements,' in 5th International Gesture Workshop, Geneva, April 2003
  4. M. J. Jones and J. M. Regh, 'Statistical color models with application to skin detection,' in Proc. Computer Vision and Pattern Recognition, pp.274-280, June 1999
  5. J. Fritsch, S. Lang, M. Kleinehagenbrock, G. A. Fink and G. Sagerer, 'Improving Adaptive Skin' Color Segmentation by Incorporating Results from Face Detection,' IEEE Int. Workshop on Robot and Human Interactive Communication, September 2002
  6. N. Tanibata, N. Shimada and Y. Shirai, 'Extraction of Hand Features for Recognition of Sign Language Wordain Proc. of Int. Conf on Vision Interface, pp.391-398, 2002
  7. N. Soontranon, S. Aramvith and T.H. Chalidabhongse, 'Face and Hand Localization and Tracking for Sign Language Recognition,' in Proc. International Symposium on Communication and Information Technologies (ISCIT'04), October 2004
  8. R. L. Hsu, M. Abdel-Mottaleb and A. K. Jain, 'Face Detection in Color Images,'IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.2, No.5, pp.696-706, May 2002