Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Deep Learning-Based Motion Reconstruction Using Tracker Sensors

트래커를 활용한 딥러닝 기반 실시간 전신 동작 복원

  • Hyunseok Kim (Dept, of Computer and Software, Hanyang University) ;
  • Kyungwon Kang (Dept, of Computer and Software, Hanyang University) ;
  • Gangrae Park (Dept, of Computer and Software, Hanyang University) ;
  • Taesoo Kwon (Dept, of Computer and Software, Hanyang University)
  • 김현석 (한양대학교 일반대학원 컴퓨터 소프트웨어학과) ;
  • 강경원 (한양대학교 일반대학원 컴퓨터 소프트웨어학과) ;
  • 박강래 (한양대학교 일반대학원 컴퓨터 소프트웨어학과) ;
  • 권태수 (한양대학교 일반대학원 컴퓨터 소프트웨어학과)
  • Received : 2023.10.04
  • Accepted : 2023.11.10
  • Published : 2023.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a novel deep learning-based motion reconstruction approach that facilitates the generation of full-body motions, including finger motions, while also enabling the online adjustment of motion generation delays. The proposed method combines the Vive Tracker with a deep learning method to achieve more accurate motion reconstruction while effectively mitigating foot skating issues through the use of an Inverse Kinematics (IK) solver. The proposed method utilizes a trained AutoEncoder to reconstruct character body motions using tracker data in real-time while offering the flexibility to adjust motion generation delays as needed. To generate hand motions suitable for the reconstructed body motion, we employ a Fully Connected Network (FCN). By combining the reconstructed body motion from the AutoEncoder with the hand motions generated by the FCN, we can generate full-body motions of characters that include hand movements. In order to alleviate foot skating issues in motions generated by deep learning-based methods, we use an IK solver. By setting the trackers located near the character's feet as end-effectors for the IK solver, our method precisely controls and corrects the character's foot movements, thereby enhancing the overall accuracy of the generated motions. Through experiments, we validate the accuracy of motion generation in the proposed deep learning-based motion reconstruction scheme, as well as the ability to adjust latency based on user input. Additionally, we assess the correction performance by comparing motions with the IK solver applied to those without it, focusing particularly on how it addresses the foot skating issue in the generated full-body motions.

본 논문에서는 손 동작을 포함한 전신 동작 생성이 가능하고 동작 생성 딜레이를 조절할 수 있는 새로운 딥러닝 기반 동작 복원 기술을 제안한다. 제안된 방법은 범용적으로 사용되는 센서인 바이브 트래커와 딥러닝 기술의 융합을 통해 더욱 정교한 동작 복원을 가능하게함과 동시에 IK 솔버(Inverse Kinematics solver)를 활용하여 발 미끄러짐 현상을 효과적으로 완화한다. 본 논문은 학습된 오토인코더(AutoEncoder)를 사용하여 트래커 데이터에 적절한 캐릭터 동작의 실시간 복원이 가능하고, 동작 복원 딜레이를 조절할 수 있는 방법을 제안한다. 복원된 전신 동작에 적합한 손 동작을 생성하기 위해 FCN(Fully Connected Network)을 사용하여 손 동작을 생성하고, 오토인코더에서 복원된 전신 동작과 FCN 에서 생성된 손 동작을 합쳐 손 동작이 포함된 캐릭터의 전신 동작을 생성할 수 있다. 앞서 딥러닝 기반의 방법으로 생성된 동작에서 발 미끄러짐 현상을 완화시키기 위해 본 논문에서는 IK 솔버 를 활용한다. 캐릭터의 발에 위치한 트래커를 IK 솔버의 엔드이펙터(end-effector)로 설정하여 캐릭터의 발 움직임을 정확하게 제어하고 보정하는 기술을 제안함으로써, 생성된 동작의 전반적인 정확성을 향상시켜 고품질의 동작을 생성한다. 실험을 통해, 본 논문에서 제안한 딥러닝 기반 동작 복원에서 정확한 동작 생성과 사용자 입력에 따라 프레임 딜레이 조정이 가능함을 검증하였고, 생성된 전신 동작의 발미끄러짐 현상에 대해 IK 솔버가 적용되기 이전 전신 동작과 비교하여 보정에 대한 성능을 확인하였다.

Keywords

Acknowledgement

이 논문은 2023년도 정부(과학기술정보통신부)의 재원으로 정보통신기획평가원의 지원(No.2020-0-01373, 인공지능대학원지원(한양대학교))과 2023년도 패러블엔터테인먼트의 지원을 받아 수행된 연구임 (1425179536, 메타버스 콘텐츠 제작을 위한 올인원 프로그램 개발)

References

  1. T. Sweeney, "Foundational principles & technologies for the metaverse," in ACM SIGGRAPH 2019 Talks, SIG-GRAPH '19, (New York, NY, USA), Association for Computing Machinery, 2019. 
  2. A. Chatzitofis, G. Albanis, N. Zioulis, and S. Thermos, "A low-cost realtime motion capture system," in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21421-21426, 2022. 
  3. J. Kim, D. Kang, Y. Lee, and T. Kwon, "Real-time interactive animation system for low-priced motion capture sensors," Journal of the Korea Computer Graphics Society, vol. 28, no. 2, pp. 29-41, 2022.  https://doi.org/10.15701/kcgs.2022.28.2.29
  4. B. Van Hooren, N. Pecasse, K. Meijer, and J. M. N. Essers, "The accuracy of markerless motion capture combined with computer vision techniques for measuring running kinematics," Scandinavian Journal of Medicine & Science in Sports, vol. 33, no. 6, pp. 966-978, 2023.  https://doi.org/10.1111/sms.14319
  5. S. L. Colyer, M. Evans, D. P. Cosker, and A. I. T. Salo, "A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system," Sports Medicine - Open, 2018/06/05. 
  6. A. Shafaei and J. Little, "Real-time human motion capture with multiple depth cameras," 2016 13th Conference on Computer and Robot Vision (CRV), pp. 24-31, 2016. 
  7. D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt, "Monocular 3d human pose estimation in the wild using improved cnn supervision," in 3D Vision (3DV), 2017 Fifth International Conference on, IEEE, 2017. 
  8. Y. Zou, J. Yang, D. Ceylan, J. Zhang, F. Perazzi, and J.-B. Huang, "Reducing footskate in human motion reconstruction with ground contact constraints," 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 448-457, 2020. 
  9. Y. Lu, H. Yu, W. Ni, and L. Song, "3d real-time human reconstruction with a single rgbd camera," Applied Intelligence, vol. 53, p. 8735-8745, aug 2022.  https://doi.org/10.1007/s10489-022-03969-4
  10. P. Caserman, A. Garcia-Agundez, and S. Goebel, "A survey of full-body motion reconstruction in immersive virtual reality applications," IEEE Transactions on Visualization and Computer Graphics, vol. 26, pp. 3089-3108, 2020.  https://doi.org/10.1109/TVCG.2019.2912607
  11. Y. Huang, M. Kaufmann, E. Aksan, M. J. Black, O. Hilliges, and G. Pons-Moll, "Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real ACM Trans. Graph., vol. 37, dec 2018. 
  12. X. Yi, Y. Zhou, and F. Xu, "Transpose: Real-time 3d human translation and pose estimation with six inertial sensors," ACM Trans. Graph., vol. 40, jul 2021. 
  13. M. Kim and S. Lee, "Fusion poser: 3d human pose estimation using sparse imus and head trackers in real time," Sensors, vol. 22, no. 13, 2022. 
  14. D. Yang, D. Kim, and S.-H. Lee, "Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals," Computer Graphics Forum, vol. 40, 2021. 
  15. K. Ahuja, E. Ofek, M. Gonzalez-Franco, C. Holz, and A. D. Wilson, "Coolmoves: User motion accentuation in virtual reality," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 5, jun 2021. 
  16. X. Yi, Y. Zhou, M. Habermann, S. Shimada, V. Golyanik, C. Theobalt, and F. Xu, "Physical inertial poser (pip): Physics-aware real-time human motion tracking from sparse inertial sensors," 2022. 
  17. A. Winkler, J. Won, and Y. Ye, "Questsim: Human motion tracking from sparse sensors with simulated avatars," SA '22, (New York, NY, USA), Association for Computing Machinery, 2022. 
  18. F. G. Harvey,M. Yurick, D. Nowrouzezahrai, and C. Pal, "Robust motion in-betweening," vol. 39, no. 4, 2020. 
  19. Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, "On the continuity of rotation representations in neural networks, 2020. 
  20. D. Bank, N. Koenigstein, and R. Giryes, "Autoencoders," 2021. 
  21. L. Kovar, J. Schreiner, and M. Gleicher, "Footskate cleanup for motion capture editing," in Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA '02, (New York, NY, USA), p. 97-104, Association for Computing Machinery, 2002. 
  22. H. Zhang, S. Starke, T. Komura, and J. Saito, "Mode-adaptive neural networks for quadruped motion control," ACM Trans. Graph., vol. 37, jul 2018.