Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of Sandcastle Play Guide Application using Artificial Intelligence and Augmented Reality

인공지능과 증강현실 기술을 이용한 모래성 놀이 가이드 애플리케이션 설계 및 구현

  • 류지승 (숭실대학교 글로벌미디어학부) ;
  • 장승우 (숭실대학교 글로벌미디어학부) ;
  • 문유정 (숭실대학교 글로벌미디어학부) ;
  • 이정진 (숭실대학교 글로벌미디어학부)
  • Received : 2022.06.18
  • Accepted : 2022.07.06
  • Published : 2022.07.26
Download PDF
⟨ Previous Next ⟩

Abstract

With the popularity and the advanced graphics hardware technology of mobile devices, various mobile applications that help children with physical activities have been studied. This paper presents SandUp, a mobile application that guides the play of building sand castles using artificial intelligence and augmented reality(AR) technology. In the process of building the sandcastle, children can interactively explore the target virtual sandcastle through the smartphone display using AR technology. In addition, to help children complete the sandcastle, SandUp informs the sand shape and task required step by step and provides visual and auditory feedback while recognizing progress in real-time using the phone's camera and deep learning classification. We prototyped our SandUp app using Flutter and TensorFlow Lite. To evaluate the usability and effectiveness of the proposed SandUp, we conducted a questionnaire survey on 50 adults and a user study on 20 children aged 4~7 years. The survey results showed that SandUp effectively helps build the sandcastle with proper interactive guidance. Based on the results from the user study on children and feedback from their parents, we also derived usability issues that can be further improved and suggested future research directions.

최근 스마트폰이 널리 보급되고 모바일 기기의 그래픽스 처리 성능이 발전함에 따라 아이들의 물리적인 활동을 돕는 다양한 모바일 애플리케이션들이 연구되고 있다. 본 논문에서는 인공지능과 증강현실 기술을 활용해 모래성 쌓기 놀이를 안내하는 모바일 애플리케이션 SandUp을 제안한다. 모래성을 쌓는 과정에서 아이는 모바일 증강현실 기술을 활용해 제시된 목표 모래성을 현실 세계에 증강하여 살펴볼 수 있다. 또한, SandUp은 모래성의 완성을 돕기 위해 단계적으로 필요한 모래 모양과 Task를 알려주고, 모바일 폰의 카메라와 딥러닝 인식모델을 이용해 실시간으로 현재 진행 상황을 인식하고 시각적, 청각적 피드백을 제공한다. 우리는 Flutter와 TensorFlow Lite를 이용해 SandUp 앱의 프로토타입을 구현하였다. 제안하는 SandUp 앱의 사용성과 효과를 평가하기 위해 성인을 대상으로 설문조사를 수행하고 앱이 목표로 한 4-7세 아이들을 모집하여 실험을 진행했다. 실험 결과와 학부모의 피드백을 분석하여 앱의 발전 가능성 및 개선점을 도출하고 향후 연구 방향을 제시한다.

Keywords

Acknowledgement

이 성과는 2022년 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임(No. NRF-2021R1C1C1014153).

References

  1. 조윤식, 조세홍, 김진모, et al., "모바일 플랫폼 교육 콘텐츠 지원을 위한 손 글씨 기반 텍스트 인터페이스," Journal of the Korea Computer Graphics Society, vol. 27, no. 5, pp. 81-89, 2021. https://doi.org/10.15701/kcgs.2021.27.5.81
  2. R. Skiada, E. Soroniati, A. Gardeli, and D. Zissis, "Easylexia: A mobile application for children with learning difficulties," Procedia Computer Science, vol. 27, pp. 218-228, 2014. https://doi.org/10.1016/j.procs.2014.02.025
  3. E. Cieza and D. Lujan, "Educational mobile application of augmented reality based on markers to improve the learning of vowel usage and numbers for children of a kindergarten in trujillo," Procedia computer science, vol. 130, pp. 352-358, 2018. https://doi.org/10.1016/j.procs.2018.04.051
  4. M. R. Crawford, M. D. Holder, and B. P. O'Connor, "Using mobile technology to engage children with nature," Environment and Behavior, vol. 49, no. 9, pp. 959-984, 2017. https://doi.org/10.1177/0013916516673870
  5. R. Aburukba, F. Aloul, A. Mahmoud, K. Kamili, and S. Ajmal, "Autiaid: A learning mobile application for autistic children," in 2017 IEEE 19th International Conference on eHealth Networking, Applications and Services (Healthcom). IEEE, 2017, pp. 1-6.
  6. G. Revelle and E. Reardon, "Designing and testing mobile interfaces for children," in Proceedings of the 8th International Conference on Interaction Design and Children, 2009, pp. 329-332.
  7. 강태석, 이동연, 김진모, et al., "Clo 3d 와 vuforia 를 활용한 증강현실 기반 디지털 패션 콘텐츠 제작," Journal of the Korea Computer Graphics Society, vol. 26, no. 3, pp. 21-29, 2020. https://doi.org/10.15701/kcgs.2020.26.3.21
  8. M. Tan and Q. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," in International conference on machine learning. PMLR, 2019, pp. 6105-6114.
  9. D. Chatzopoulos, C. Bermejo, Z. Huang, and P. Hui, "Mobile augmented reality survey: From where we are to where we go," Ieee Access, vol. 5, pp. 6917-6950, 2017. https://doi.org/10.1109/ACCESS.2017.2698164
  10. Google LLC. (2022) ARCore. [Online]. Available: https://developers.google.com/ar
  11. Apple Inc. (2022) ARKit. [Online]. Available: https://developer.apple.com/kr/augmented-reality/arkit/
  12. D. Nincarean, M. B. Alia, N. D. A. Halim, and M. H. A. Rahman, "Mobile augmented reality: The potential for education," Procedia-social and behavioral sciences, vol. 103, pp. 657-664, 2013. https://doi.org/10.1016/j.sbspro.2013.10.385
  13. 장현정 and 이진희, "생활주제에 따른 모래놀이가 유아의 또래 유능성 및 만족지연능력에 미치는 효과," 어린이미디어연구, vol. 12, no. 1, pp. 97-117, 2013.
  14. 김태영, "모래놀이치료가 adhd 성인의 우울 및 불안에 미치는 효과," 상징과 모래놀이치료, vol. 1, no. 1, pp. 15-40, 2010.
  15. Google LLC. (2022) Flutter. [Online]. Available: https://flutter.dev/
  16. S. A. Hassan, T. Rahim, and S. Y. Shin, "Childar: an augmented reality-based interactive game for assisting children in their education," Universal Access in the Information Society, pp. 1-12, 2021.
  17. F. Khalid, A. I. Ali, R. R. Ali, and M. S. Bhatti, "Ared: Anatomy learning using augmented reality application," in 2019 International Conference on Engineering and Emerging Technologies (ICEET). IEEE, 2019, pp. 1-6.
  18. S. N. Kundu, N. Muhammad, and F. Sattar, "Using the augmented reality sandbox for advanced learning in geoscience education," in 2017 IEEE 6th international conference on teaching, assessment, and learning for engineering (TALE). IEEE, 2017, pp. 13-17.
  19. T. V. Do and J.-W. Lee, "A multiple-level 3d-lego game in augmented reality for improving spatial ability," in International Conference on Human-Computer Interaction. Springer, 2009, pp. 296-303.
  20. W. Yan, "Augmented reality applied to lego construction: Ar-based building instructions with high accuracy & precision and realistic object-hand occlusions," arXiv preprint arXiv:1907.12549, 2019.
  21. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510-4520.
  22. J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.
  23. L. Carius, C. Eichhorn, D. A. Plecher, and G. Klinker, "Cloud-based cross-platform collaborative ar in flutter," in 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 2022, pp. 682-683.
  24. J. Li, E. D. Van der Spek, J. Hu, and L. Feijs, "Turning your book into a game: improving motivation through tangible interaction and diegetic feedback in an ar mathematics game for children," in Proceedings of the annual symposium on computer-human interaction in play, 2019, pp. 73-85.
  25. E. L. Deci and R. M. Ryan, Intrinsic motivation and self-determination in human behavior. Springer Science & Business Media, 2013.
  26. R. M. Ryan and E. L. Deci, "Intrinsic and extrinsic motivations: Classic definitions and new directions," Contemporary educational psychology, vol. 25, no. 1, pp. 54-67, 2000. https://doi.org/10.1006/ceps.1999.1020
  27. R. M. Ryan and E. L. Deci, "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being." American psychologist, vol. 55, no. 1, p. 68, 2000. https://doi.org/10.1037/0003-066X.55.1.68
  28. S. Deterding, "The lens of intrinsic skill atoms: A method for gameful design," Human-Computer Interaction, vol. 30, no. 3-4, pp. 294-335, 2015. https://doi.org/10.1080/07370024.2014.993471
  29. R. M. Ryan, "Psychological needs and the facilitation of integrative processes," Journal of personality, vol. 63, no. 3, pp. 397-427, 1995. https://doi.org/10.1111/j.1467-6494.1995.tb00501.x
  30. R. M. Ryan, E. L. Deci, et al., "Overview of self-determination theory: An organismic dialectical perspective," Handbook of self-determination research, vol. 2, pp. 3-33, 2002.
  31. M. Sailer, J. U. Hense, S. K. Mayr, and H. Mandl, "How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction," Computers in human behavior, vol. 69, pp. 371-380, 2017. https://doi.org/10.1016/j.chb.2016.12.033
  32. R. M. Ryan, C. S. Rigby, and A. Przybylski, "The motivational pull of video games: A self-determination theory approach," Motivation and emotion, vol. 30, no. 4, pp. 344-360, 2006. https://doi.org/10.1007/s11031-006-9051-8
  33. E. McAuley, T. Duncan, and V. V. Tammen, "Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis," Research quarterly for exercise and sport, vol. 60, no. 1, pp. 48-58, 1989. https://doi.org/10.1080/02701367.1989.10607413
  34. S. Rigby and R. Ryan, "The player experience of need satisfaction (pens) model," Immersyve Inc, pp. 1-22, 2007.
  35. 홍승현, 나기리, 조윤식, 김진모, et al., "모바일 가상현실에서의 이동 인터페이스에 관한 연구," Journal of the Korea Computer Graphics Society, vol. 27, no. 3, pp. 55-63, 2021.