Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Design of Handwriting-based Text Interface for Support of Mobile Platform Education Contents

모바일 플랫폼 교육 콘텐츠 지원을 위한 손 글씨 기반 텍스트 인터페이스 설계

  • Cho, Yunsik (Department of Computer Engineering, Graduate School, Hansung University) ;
  • Cho, Sae-Hong (Division of Computer Engineering, Hansung University) ;
  • Kim, Jinmo (Division of Computer Engineering, Hansung University)
  • 조윤식 (한성대학교 일반대학원 컴퓨터공학과) ;
  • 조세홍 (한성대학교 컴퓨터공학부) ;
  • 김진모 (한성대학교 컴퓨터공학부)
  • Received : 2021.11.19
  • Accepted : 2021.11.26
  • Published : 2021.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

This study proposes a text interface for support of language-based educational contents in a mobile platform environment. The proposed interface utilizes deep learning as an input structure to write words through handwriting. Based on GUI (Graphical User Interface) using buttons and menus of mobile platform contents and input methods such as screen touch, click, and drag, we design a text interface that can directly input and process handwriting from the user. It uses the EMNIST (Extended Modified National Institute of Standards and Technology database) dataset and a trained CNN (Convolutional Neural Network) to classify and combine alphabetic texts to complete words. Finally, we conduct experiments to analyze the learning support effect of the interface proposed by directly producing English word education contents and to compare satisfaction. We compared the ability to learn English words presented by users who have experienced the existing keypad-type interface and the proposed handwriting-based text interface in the same educational environment, and we analyzed the overall satisfaction in the process of writing words by manipulating the interface.

본 연구는 모바일 플랫폼 환경에서 언어 기반의 교육 콘텐츠 지원을 위한 텍스트 인터페이스를 제안한다. 이는 손 글씨를 통해 단어를 작성하는 입력 구조로 딥 러닝을 활용한다. 모바일 플랫폼 콘텐츠의 버튼, 메뉴 등을 활용한 GUI (Graphical User Interface)와 화면 터치, 클릭, 드래그 등의 입력 방식을 기반으로 손 글씨를 사용자로부터 직접 입력하여 처리할 수 있는 텍스트 인터페이스를 설계한다. 이는 EMNIST (Extended Modified National Institute of Standards and Technology database) 데이터 셋과 훈련된 CNN (Convolutional Neural Network)을 사용하여 알파벳 텍스트를 분류하고 조합하여 단어를 완성한다. 최종적으로 영어 단어 교육 콘텐츠를 직접 제작하여 제안하는 인터페이스의 학습 지원 효과를 분석하고 만족도를 비교하기 위한 실험을 진행한다. 동일한 교육 환경에서 기존의 키 패드 방식의 인터페이스와 제안하는 손 글씨 기반 텍스트 인터페이스를 서로 체험한 사용자들이 제시하는 영어 단어를 학습하는 능력을 비교하고, 인터페이스를 조작하여 단어를 작성하는 과정에서의 전체적인 만족도를 분석, 확인하도록 한다.

Keywords

Acknowledgement

이 논문은 2021년도 정부(과학기술정보통신부)의 재원으로 정보통신기획평가원의 지원을 받아 수행된 연구임(연구개발과제번호: 2021-0-00884, 비대면 협업용 솔루션 및 블록체인 기반 디지털 워크 통합 플랫폼 개발, 기여율 30%, 조세홍). 그리고 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임 (No. 2020R1F1A1063442, 기여율 40%, 조윤식). 또한, 본 연구는 한성대학교 학술연구비 지원과제임 (기여율 30%, 김진모).

References

  1. T. Sweeney, "Foundational principles & technologies for the metaverse," in ACM SIGGRAPH 2019 Talks, ser. SIGGRAPH '19. New York, NY, USA: Association for Computing Machinery, 28 July-1 August 2019. [Online]. Available: https://doi.org/10.1145/3306307.3339844
  2. A. van Dam, "Post-wimp user interfaces," Commun. ACM, vol. 40, no. 2, p. 63-67, feb 1997. [Online]. Available: https://doi.org/10.1145/253671.253708
  3. H.-Y. Huang, C.-W. Ning, P.-Y. Wang, J.-H. Cheng, and L.-P. Cheng, "Haptic-go-round: A surrounding platform for encounter-type haptics in virtual reality experiences," in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ser. CHI '20. New York, NY, USA: Association for Computing Machinery, 25-30 April 2020, p. 1-10. [Online]. Available: https://doi.org/10.1145/3313831.3376476
  4. Shneiderman, "Direct manipulation: A step beyond programming languages," Computer, vol. 16, no. 8, pp. 57-69, 1983. https://doi.org/10.1109/MC.1983.1654471
  5. J.-H. Sung and T. Y. Kim, "Leap motion-based playing yut through hand gesture interaction," The Journal of Korean Institute of Next Generation Computing, vol. 13, no. 1, pp. 74-81, 2017.
  6. B. D. Song, H.-J. Kim, H.-J. Jeong, and Y. J. Choi, "Development of interactive media player for kiosk with user motion detection," The Journal of the Korea Contents Association, vol. 19, no. 11, pp. 270-277, 2019. https://doi.org/10.5392/JKCA.2019.19.11.270
  7. J.-H. Kim, "Interaction technique in smoke simulations using mouth-wind on mobile devices," Journal of the Korea Computer Graphics Society, vol. 24, no. 4, pp. 21-27, 2018. https://doi.org/10.15701/kcgs.2018.24.5.21
  8. S. Hong, G. Na, Y. Cho, and J. Kim, "A study on movement interface in mobile virtual reality," Journal of the Korea Computer Graphics Society, vol. 27, no. 3, pp. 55-63, 2021.
  9. Y. Cho, J. Kang, J. Jeon, J. Park, M. Kim, and J. Kim, "X-person asymmetric interaction in virtual and augmented realities," Computer Animation and Virtual Worlds, vol. 32, no. 5, p. e1985, 2021. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/cav.1985
  10. Y. Cho and J. Kim, "Production of mobile english language teaching application based on text interface using deep learning," Electronics, vol. 10, no. 15, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/15/1809
  11. T. Kang, M. Chae, E. Seo, M. Kim, and J. Kim, "Deep-handsvr: Hand interface using deep learning in immersive virtual reality," Electronics, vol. 9, no. 11, 2020. [Online]. Available: https://www.mdpi.com/2079-9292/9/11/1863
  12. R. Jayakanthan, "Application of computer games in the field of education," Electronic Library, vol. 20, no. 2, pp. 98-102, 2002. [Online]. Available: https://www.learntechlib.org/p/95865 https://doi.org/10.1108/02640470210697471
  13. V. S. Zirawaga, A. I. Olusanya, and T. Maduku, "Gaming in education: Using games as a support tool to teach history," Journal of Education and Practice, vol. 8, no. 15, pp. 55-64, 2017.
  14. 남현숙, "2020년 국외 디지털콘텐츠 시장조사 및 동향 심층분석," 소프트웨어정책연구소 연구보고서, vol. RE-107, pp. 1-378, 2021.
  15. Y. Chen, D. Zhou, Y. Wang, and J. Yu, "Application of augmented reality for early childhood english teaching," in 2017 International Symposium on Educational Technology (ISET), 2017, pp. 111-115.
  16. M. Fan, A. N. Antle, and J. L. Warren, "Augmented reality for early language learning: A systematic review of augmented reality application design, instructional strategies, and evaluation outcomes," Journal of Educational Computing Research, vol. 58, no. 6, pp. 1059-1100, 2020. [Online]. Available: https://doi.org/10.1177/0735633120927489
  17. M. Lu, "Effectiveness of vocabulary learning via mobile phone," Journal of Computer Assisted Learning, vol. 24, no. 6, pp. 515-525, 2008. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2729.2008.00289.x
  18. J. Jun, J. Lim, and S. Kim, "Investigating the educational effectiveness of an online educational game," The journal of Educational Studies, vol. 39, no. 1, pp. 85-105, 2008.
  19. D. Aminatun and L. Oktaviani, "Memrise: Promoting students' autonomous learning skill through language learning application," Metathesis: Journal of English Language, Literature, and Teaching, vol. 3, no. 2, pp. 214-223, 2019. https://doi.org/10.31002/metathesis.v3i2.1982
  20. S. E. Lindley, J. Le Couteur, and N. L. Berthouze, "Stirring up experience through movement in game play: Effects on engagement and social behaviour," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI '08. New York, NY, USA: Association for Computing Machinery, 2008, p. 511-514. [Online]. Available: https://doi.org/10.1145/1357054.1357136
  21. K. V. Nesbitt and I. Hoskens, "Multi-sensory game interface improves player satisfaction but not performance," in Proceedings of the Ninth Conference on Australasian User Interface - Volume 76, ser. AUIC '08. AUS: Australian Computer Society, Inc., 2008, p. 13-18.
  22. N. Sidorakis, G. A. Koulieris, and K. Mania, "Binocular eye-tracking for the control of a 3d immersive multimedia user interface," in 2015 IEEE 1st Workshop on Everyday Virtual Reality (WEVR), 23 March 2015, pp. 15-18.
  23. M. Kim, J. Lee, C. Jeon, and J. Kim, "A study on interaction of gaze-based user interface in mobile virtual reality environment," Journal of the Korea Computer Graphics Society, vol. 23, no. 3, pp. 39-46, 2017. [Online]. Available: http://dx.doi.org/10.15701/kcgs.2017.23.3.39
  24. I. Choi, E. Ofek, H. Benko, M. Sinclair, and C. Holz, "Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI '18. New York, NY, USA: ACM, 21-26 April 2018, pp. 654:1-654:13. [Online]. Available: http://doi.acm.org/10.1145/3173574.3174228
  25. J. Kim, "A study on comparative experiment of hand-based interface in immersive virtua reality," Journal of the Korea Computer Graphics Society, vol. 25, no. 2, pp. 1-9, 2019.
  26. M. Kim, J. Kim, K. Jeong, and C. Kim, "Grasping vr: Presence of pseudo-haptic interface based portable hand grip system in immersive virtual reality," International Journal of Human-Computer Interaction, vol. 36, no. 7, pp. 685-698, 2020. https://doi.org/10.1080/10447318.2019.1680920
  27. J. Kim, "Vivr: Presence of immersive interaction for visual impairment virtual reality," IEEE Access, vol. 8, pp. 196 151-196 159, 2020. https://doi.org/10.1109/access.2020.3034363
  28. S. Yang and M. Zhang, "Application of brain neural network in personalized english education system," International Journal of Emerging Technologies in Learning (iJET), vol. 13, no. 10, pp. 15-22, 2018. [Online]. Available: https://online-journals.org/index.php/i-jet/article/view/9488
  29. G. Cohen, S. Afshar, J. Tapson, and A. van Schaik, "Emnist: Extending mnist to handwritten letters," in 2017 International Joint Conference on Neural Networks (IJCNN), 14-19 May 2017, pp. 2921-2926.
  30. A. Baldominos, Y. Saez, and P. Isasi, "A survey of handwritten character recognition with mnist and emnist," Applied Sciences, vol. 9, no. 15, 2019. [Online]. Available: https://www.mdpi.com/2076-3417/9/15/3169
  31. S. S. Mor, S. Solanki, S. Gupta, S. Dhingra, M. Jain, and R. Saxena, "Handwritten text recognition: With deep learning and android," International Journal of Engineering and Advanced Technology (IJEAT), vol. 8, no. 3S, pp. 819-825, 2019. [Online]. Available: https://www.ijeat.org/wpcontent/uploads/papers/v8i3S/C11730283S19.pdf https://doi.org/10.35940/ijeat.F1159.0986S319
  32. UnityTechnologies, "Unity engine," Unity Technologies, 2019. [Online]. Available: https://unity3d.com/
  33. GoogleBrain, "Tensorflow," Google Brain, 2021. [Online]. Available: https://www.tensorflow.org/
  34. A. Lund, "Measuring usability with the use questionnaire," Usability Interface, vol. 8, no. 2, pp. 3-6, 01 2001.