The Journal of the Institute of Internet, Broadcasting and Communication (한국인터넷방송통신학회논문지)

The Institute of Internet, Broadcasting and Communication (한국인터넷방송통신학회)
권호 표지
DOI QR코드

DOI QR Code

Development of Automatic Lip-sync MAYA Plug-in for 3D Characters

3D 캐릭터에서의 자동 립싱크 MAYA 플러그인 개발

  • 이상우 (한국산업기술대학교, 컴퓨터공학과) ;
  • 신성욱 (한국산업기술대학교, 컴퓨터공학과) ;
  • 정성택 (한국산업기술대학교, 컴퓨터공학과)
  • Received : 2018.05.21
  • Accepted : 2018.06.08
  • Published : 2018.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we have developed the Auto Lip-Sync Maya plug-in for extracting Korean phonemes from voice data and text information based on Korean and produce high quality 3D lip-sync animation using divided phonemes. In the developed system, phoneme separation was classified into 8 vowels and 13 consonants used in Korean, referring to 49 phonemes provided by Microsoft Speech API engine SAPI. In addition, the pronunciation of vowels and consonants has variety Mouth Shapes, but the same Viseme can be applied to some identical ones. Based on this, we have developed Auto Lip-sync Maya Plug-in based on Python to enable lip-sync animation to be implemented automatically at once.

본 논문에서는 한국어를 기반으로 음성 데이터와 텍스트 정보에서 한국어 음소를 추출하고 분할된 음소들을 사용하여 정확하고 자연스러운 3D 립싱크 애니메이션을 제작하기 위한 오토 립싱크 Maya 플러그인을 개발하였다. 여기서 개발된 시스템에서는 음소 분할은 Microsoft Speech API 엔진 SAPI에서 제공하는 49개의 음소를 참조하여 한글에 사용되는 음소들을 모음 8개, 자음 13개로 분류하였다. 또한 모음과 자음의 발음들은 다양한 입모양을 가지지만 일부 동일한 입모양에 대하여 같은 Viseme을 적용할 수 있도록 구현하였다. 이를 바탕으로 파이썬(Python) 기반의 오토 립싱크 Maya 플러그인을 개발하여 립싱크 애니메이션이 한 번에 자동으로 구현할 수 있게 하였다.

Keywords

References

  1. Yong Wook Kim, "Market Analysis & Industrial Trends of Cartoon.Game.Animation Contents Industry", Review of Korea Contents Association, Vol. 13, No. 2, pp. 14-19, June 2015.
  2. Ji-Ae Kim, "A Study on Effective Facial Expression of 3D Character through Variation of Emotions - Model using Facial Anatomy", The Journal of Korea Multimedia Society, Vol. 9, No. 7, pp. 894-903, July 2006.
  3. https://www.kocca.kr/
  4. Chul Hee Jung, Mingeun Lee, and Myeong Won Lee, "Development of Exchangeable Character Animation Using a Motion Sensor", Korean Society For Computer Game, Vol. 27, No. 4, pp. 237-246, Dec. 2014.
  5. Youn Hwan Ha and Sung Hong Hong, "Recognizing Human Facial Expressions and Gesture from Image Sequence", Journal of Biomedical Engineering Research, Vol. 20, No. 4, pp. 419-425, 1999.
  6. Yu-Sin Park, Ko-Rock Sin, Tae-Yong Kim, and Jong Soo Cho, "Technical Analysis of Text to Lip Sync. Animation", Korean Society For Computer Game, Vol. 1, No.2, pp. 36-45, June 2003.
  7. Hyewon Song, Suwoong Heo, Jiwoo Kang, and Sanghoon Lee, "3D Character Animation: A Brief Review", Journal of International Society for Simulation Surgery, Vol. 2, No. 2, pp. 52-57, Dec. 2015. DOI: http://dx.doi.org/10.18204/JISSiS.2015.2.2.052
  8. Dong Sun Shin and Jinoh Chung, "A Study on 3D Graphics Design System for the Korean Lip-Sync Synthesis", Proceeding of HCI Korea 2016. pp. 1762-1769, 2016.
  9. John Lasseter, "Principles of Traditional Animation Applied to 3D Computer Animation", Computer Graphics, Vol. 21, No. 4, pp. 35-44, July 1987. DOI: http://dx.doi.org/10.1145/37401.37407
  10. Yongqiang Li and Shangfei Wang, "Simultaneous Facial Feature Tracking and Facial Expression Recognition", IEEE Transaction on Image Processing. Vol. 22, No. 7, pp. 2559-2573, July 2013. DOI: http://dx.doi.org/10.1109/TIP.2013.2253477
  11. Seong-Ho Lee, Kyong-Ho Han, "Detection of Moving Objects using Depth Frame Data of 3D Sensor", The Journal of The Institute of Internet, Broadcasting and Communication(JIIBC), Vol. 14, No. 5, pp. 243-248, Oct. 2014. DOI: http://dx.doi.org/10.7236/JIIBC.2014.14.5.243
  12. Min-Sung Kim, "A Study on Overcome of Marker-based Motion Capture Environment", Journal of the Korea Entertainment Industry Association, Vol. 10, No. 2, pp. 17-25, Apr. 2016. DOI : http://dx.doi.org/10.21184/jkeia.2016.04.10.2.17
  13. https://msdn.microsoft.com/en-us/library/ms717037(v=vs.85).aspx
  14. https://github.com/DanielSWolf/rhubarb-lip-sync.
  15. http://www.garycmartin.com/mouth_shapes.html
  16. Il-Hong Jung, Eun-Ji Kim,"Natural 3D Lip-Synch Animation Based on Korean Phonemic Data", Journal of Digital Contents Society, Vol. 9, No. 2, pp. 331-339, June. 2008.