Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Motion Generation of a Single Rigid Body Character Using Deep Reinforcement Learning

심층 강화 학습을 활용한 단일 강체 캐릭터의 모션 생성

  • Ahn, Jewon (Dept. of Intelligence Convergence, Hanyang University) ;
  • Gu, Taehong (Dept. of Computer and Software, Hanyang University) ;
  • Kwon, Taesoo (Dept. of Computer and Software, Hanyang University)
  • 안제원 (한양대학교 지능융합학과) ;
  • 구태홍 (한양대학교 컴퓨터 소프트웨어학과) ;
  • 권태수 (한양대학교 컴퓨터 소프트웨어학과)
  • Received : 2021.06.09
  • Accepted : 2021.06.25
  • Published : 2021.07.23
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we proposed a framework that generates the trajectory of a single rigid body based on its COM configuration and contact pose. Because we use a smaller input dimension than when we use a full body state, we can improve the learning time for reinforcement learning. Even with a 68% reduction in learning time (approximately two hours), the character trained by our network is more robust to external perturbations tolerating an external force of 1500 N which is about 7.5 times larger than the maximum magnitude from a previous approach. For this framework, we use centroidal dynamics to calculate the next configuration of the COM, and use reinforcement learning for obtaining a policy that gives us parameters for controlling the contact positions and forces.

본 논문에서는 단일 강체 모델(single rigid body)의 무게 중심(center of mass) 좌표계와 발의 위치를 활용하여 캐릭터의 동작을 생성하는 프레임워크를 제안한다. 이 프레임워크를 사용하면 기존의 전신 동작(full body)에 대한 정보를 사용할 때 보다 입력 상태 벡터(input state)의 차원을 줄임으로써 강화 학습의 속도를 개선할 수 있다. 또한 기존의 방법보다 학습 속도를 약 2 시간(약 68% 감소) 감소시켰음에도 기존의 방법 대비 최대 7.5배(약 1500 N)의 외력을 더 견딜 수 있는 더욱 견고한(robust) 모션을 생성할 수 있다. 본 논문에서는 이를 위해 무게 중심의 다음 좌표계를 구하기 위해 중심 역학(centroidal dynamics)을 활용하였고, 이에 필요한 매개 변수(parameter)들과 다음 발의 위치와 접촉력 계산에 필요한 매개 변수들을 구하는 정책(policy)의 학습을 심층 강화 학습(deep reinforcement learning)을 사용하여 구현하였다.

Keywords

References

  1. Peng, Xue Bin, et al. "Deepmimic: Example-guided deep reinforcement learning of physics-based character skills." ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1-14, 2018.
  2. Coros, Stelian, Philippe Beaudoin, and Michiel Van de Panne. "Generalized biped walking control." ACM Transactions on Graphics (TOG), vol. 29, no. 4, pp. 1-9, 2010.
  3. Ye, Yuting, and C. Karen Liu. "Optimal feedback control for character animation using an abstract model." ACM SIGGRAPH 2010 papers, pp. 1-9, 2010.
  4. Yin, KangKang, Kevin Loken, and Michiel Van de Panne. "Simbicon: Simple biped locomotion control." ACM Transactions on Graphics(TOG), vol. 26, no. 3, pp. 105-es, 2007. https://doi.org/10.1145/1276377.1276509
  5. Agrawal, Shailen, Shuo Shen, and Michiel van de Panne. "Diverse motion variations for physics-based character animation." Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 37-44, 2013.
  6. Ha, Sehoon, and C. Karen Liu. "Iterative training of dynamic skills inspired by human coaching techniques." ACM Transactions on Graphics (TOG), vol. 34, no. 1, pp. 1-11, 2014.
  7. Wang, Jack M., et al. "Optimizing locomotion controllers using biologically-based actuators and objectives." ACM Transactions on Graphics (TOG), vol. 31, no. 4, pp. 1-11, 2012.
  8. Da Silva, Marco, Yeuhi Abe, and Jovan Popovic. "Simulation of human motion data using short-horizon model-predictive control." Computer Graphics Forum. Oxford, UK. Blackwell Publishing Ltd, vol. 27, no. 2, pp. 371-380, 2008.
  9. Lee, Yoonsang, Sungeun Kim, and Jehee Lee. "Data-driven biped control." ACM SIGGRAPH 2010 papers, pp. 1-8, 2010.
  10. Lee, Yoonsang, et al. "Locomotion control for many-muscle humanoids." ACM Transactions on Graphics (TOG), vol. 33, no. 6, pp. 1-11, 2014.
  11. Mordatch, Igor, Emanuel Todorov, and Zoran Popovic. "Discovery of complex behaviors through contact-invariant optimization." ACM Transactions on Graphics (TOG), vol. 31, no. 4, pp. 1-8, 2012.
  12. Wampler, Kevin, Zoran Popovic, and Jovan Popovic. "Generalizing locomotion style to new animals with inverse optimal regression." ACM Transactions on Graphics (TOG), vol. 33, no. 4, pp. 1-11, 2014.
  13. Hamalainen, Perttu, Joose Rajamaki, and C. Karen Liu. "Online control of simulated humanoids using particle belief propagation." ACM Transactions on Graphics (TOG), vol. 34, no. 4, pp. 1-13, 2015.
  14. Tassa, Yuval, Tom Erez, and Emanuel Todorov. "Synthesis and stabilization of complex behaviors through online trajectory optimization." 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 4906-4913, 2012.
  15. Lee, Yongjoon, et al. "Motion fields for interactive character locomotion." ACM SIGGRAPH Asia 2010 papers, pp. 1-8, 2010.
  16. Levine, Sergey, et al. "Continuous character control with low-dimensional embeddings." ACM Transactions on Graphics (TOG), vol. 31, no. 4, pp. 1-10, 2012. https://doi.org/10.1145/2185520.2185524
  17. Coros, Stelian, Philippe Beaudoin, and Michiel Van de Panne. "Robust task-based control policies for physics-based characters." ACM SIGGRAPH Asia 2009 papers, pp. 1-9, 2009.
  18. Peng, Xue Bin, Glen Berseth, and Michiel Van de Panne. "Dynamic terrain traversal skills using reinforcement learning." ACM Transactions on Graphics (TOG), vol. 34, no. 4, pp. 1-11, 2015.
  19. Brockman, Greg, et al. "Openai gym." arXiv preprint arXiv:1606.01540, 2016.
  20. Duan, Yan, et al. "Benchmarking deep reinforcement learning for continuous control." International conference on machine learning. PMLR, pp. 1329-1338, 2016.
  21. Liu, Libin, and Jessica Hodgins. "Learning to schedule control fragments for physics-based characters using deep q-learning." ACM Transactions on Graphics (TOG), vol. 36, no. 3, pp. 1-14, 2017.
  22. Peng, Xue Bin, Glen Berseth, and Michiel Van de Panne. "Terrain-adaptive locomotion skills using deep reinforcement learning." ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1-12, 2016.
  23. Rajeswaran, Aravind, et al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations." arXiv preprint arXiv:1709.10087, 2017.
  24. Teh, Yee Whye, et al. "Distral: Robust multitask reinforcement learning." arXiv preprint arXiv:1707.04175, 2017.
  25. Orin, David E., Ambarish Goswami, and Sung-Hee Lee. "Centroidal dynamics of a humanoid robot." Autonomous robots, vol. 35, no. 2, pp. 161-176, 2013. https://doi.org/10.1007/s10514-013-9341-4
  26. Dai, Hongkai, Andres Valenzuela, and Russ Tedrake. "Whole-body motion planning with centroidal dynamics and full kinematics." 2014 IEEE-RAS International Conference on Humanoid Robots. IEEE, pp. 295-302, 2014.
  27. Winkler, Alexander W., et al. "Gait and trajectory optimization for legged systems through phase-based end-effector parameterization." IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 1560-1567, 2018. https://doi.org/10.1109/lra.2018.2798285
  28. Kwon, Taesoo, Yoonsang Lee, and Michiel Van De Panne. "Fast and flexible multilegged locomotion using learned centroidal dynamics." ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 1-46, 2020.
  29. Xie, Zhaoming, et al. "GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model." arXiv preprint arXiv:2104.09771, 2021.
  30. Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347, 2017.
  31. Abe, Yeuhi, Marco Da Silva, and Jovan Popovic. "Multiobjective control with frictional contacts." Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 249-258, 2007.
  32. da Silva, Marco, Yeuhi Abe, and Jovan Popovic. "Interactive simulation of stylized human locomotion." ACM SIGGRAPH 2008 papers, pp. 1-10, 2008.
  33. Kwon, Taesoo, and Jessica K. Hodgins. "Momentum-mapped inverted pendulum models for controlling dynamic human motions." ACM Transactions on Graphics(TOG), vol. 36, no. 1, pp. 1-14, 2017.
  34. Ellis, Jane, et al. "CDM: Taking stock and looking forward." Energy policy, vol. 35, no. 1, pp. 15-28, 2007. https://doi.org/10.1016/j.enpol.2005.09.018

Cited by

  1. Research on the Application of Intelligent Choreography for Musical Theater Based on Mixture Density Network Algorithm vol.2021, 2021, https://doi.org/10.1155/2021/4337398