Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Implementation of Paper Keyboard Piano with a Kinect

키넥트를 이용한 종이건반 피아노 구현 연구

  • Lee, Jung-Chul (School of Electrical Engineering, University of Ulsan) ;
  • Kim, Min-Seong (School of Electrical Engineering, University of Ulsan)
  • Received : 2012.12.05
  • Accepted : 2012.12.20
  • Published : 2012.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a paper keyboard piano implementation using the finger movement detection with the 3D image data from a kinect. Keyboard pattern and keyboard depth information are extracted from the color image and depth image to detect the touch event on the paper keyboard and to identify the touched key. Hand region detection error is unavoidable when using the simple comparison method between input depth image and background depth image, and this error is critical in key touch detection. Skin color is used to minimize the error. And finger tips are detected using contour detection with area limit and convex hull. Finally decision of key touch is carried out with the keyboard pattern information at the finger tip position. The experimental results showed that the proposed method can detect key touch with high accuracy. Paper keyboard piano can be utilized for the easy and convenient interface for the beginner to learn playing piano with the PC-based learning software.

본 논문에서는 키넥트의 3차원 영상정보를 이용하여 손가락 움직임을 검출하고 이 정보를 이용하여 종이건반 피아노를 구현하는 방법을 제안한다. 키넥트의 컬러영상과 깊이영상을 이용하여 먼저 건반 식별을 위해 필요한 건반 패턴 정보와 사용자의 종이건반 누름 여부를 판정하기 위한 건반깊이 정보를 검출한다. 배경의 깊이정보를 이용하여 입력된 깊이정보로부터 손 영역을 검출할 경우 종이건반과 접촉한 손가락 끝부분이 잘려나가는 문제를 해결하기 위해 스킨 컬러를 이용하여 정확도를 향상시켰다. 그리고 면적을 이용한 외곽선 검출과 convex hull 알고리즘을 이용하여 손가락 끝점을 검출하고 건반 패턴 정보와 비교하여 건반 누름을 판정하였다. 본 논문에서 제안하는 방법을 이용하여 종이건반 피아노 성능평가를 수행한 결과 높은 검출 정확도를 보였다. 구현된 종이건반 피아노 기능을 활용하여 피아노 연주 입력장치를 구현함으로써 PC기반 피아노 연주시스템 구현의 편리성을 보였으며 악기 초보자가 PC 기반 피아노 연주 학습에 사용자 인터페이스로 활용할 수 있는 가능성을 확인하였다.

Keywords

References

  1. Akai EWI4000S, http://www.akaipro.com/
  2. WX5 YAMAHAMidi Controller, http://www.yamaha. com/
  3. Finale&SmartMusic, http://www.makemusic.com/
  4. P. Messic, "Maximum MIDI: Music Applications in C++," Manning Publications, 1998.
  5. K.M. Cho, J.H. Jang, and K.S. Hong, "Adaptive Skin-Color Filter," Pattern Recognition, vol.34, no.5, pp.1067-1073, May 2001. https://doi.org/10.1016/S0031-3203(00)00034-0
  6. P. Viola , M. Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", IEEE Conference on Computer Vision and Pattern Recognition, vol.1, pp.511- 518, Dec. 2001.
  7. B.M. Kim, J.W. Kim, and K.H. Lee, "An Application of AdaBoost learning Algorithm and Kalman Filter to Hand Detection and Tracking," Journal of The Korea Society of Computer and Information, vol.10, no.4, pp.47-56, Sep. 2005.
  8. J.M. Jeong, J.R. Jang, Y.I. Kim, J.W. Park, and W.J. Lee, "Development of the Hand Recognition System for the Mouse Control," Proceeding of Winter Conference of KSCI, vol.19, no.1, pp.173-174, Jan. 2011.
  9. S.W. Jang, Y.J. Park, and G.Y. Kim, "Human Skin Region Detection Utilizing Depth Information," Journal of The Korea Society of Computer and Information, vol.17, no.6, pp.29-36, June 2012. https://doi.org/10.9708/jksci.2012.17.6.029
  10. Microsoft Kinect, http://www.microsoft.com/en-us/ kinectforwindows/
  11. S.Y. Cho, H.R. Byun, H.K. Lee, and J.H. Cha, "Hand Gesture Recognition from Kinect Sensor Data," Journal of Broadcast Engineering, vol.17, no.3, pp.447-458, May 2012. https://doi.org/10.5909/JBE.2012.17.3.447
  12. S.H. Lee, H.S. Hahn, and Y.J. Han, "The Estimation of Hand Pose Based on Mean-Shift Tracking Using the Fusion of Color and Depth Information for Marker-less Augmented Reality," Journal of The Korea Society of Computer and Information, vol.17, no.7, pp.155-166, July 2012. https://doi.org/10.9708/jksci.2012.17.7.155
  13. Y. Cheng, "Mean shift, mode seeking, and clustering," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.17, no.8, pp.790-799, Aug. 1995. https://doi.org/10.1109/34.400568
  14. kinect piano download, http://www.google.co.kr/#q =kinect+piano+download&hl=ko&newwindow= 1&sa=N&tbo=u&source=univ&tbm=vid&ei=R CPQUNz8OOqaiAe1qoHAAg&ved=0CEsQqwQ4C g&bav=on.2,or.r_gc.r_pw.r_qf.&bvm=bv.135553 4169,d.aGc&fp=7605525a9eab9576&bpcl=40096 503&biw=1090&bih=1036
  15. OpenNI, http://www.openni.org/documentation/
  16. OpenNI, http://www.openni.org/Downloads/ OpenNI-Modules.aspx
  17. Xianquan Zhang and Zhenjun Tang, "A Fast Convex Hull Algorithm for Binary Image," Informatica vol.34, no.3, pp.369-376, Mar. 2010.