ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Motion Capture of the Human Body Using Multiple Depth Sensors

  • Received : 2016.08.15
  • Accepted : 2017.02.09
  • Published : 2017.04.01
Download PDF
⟨ Previous Next ⟩

Abstract

The movements of the human body are difficult to capture owing to the complexity of the three-dimensional skeleton model and occlusion problems. In this paper, we propose a motion capture system that tracks dynamic human motions in real time. Without using external markers, the proposed system adopts multiple depth sensors (Microsoft Kinect) to overcome the occlusion and body rotation problems. To combine the joint data retrieved from the multiple sensors, our calibration process samples a point cloud from depth images and unifies the coordinate systems in point clouds into a single coordinate system via the iterative closest point method. Using noisy skeletal data from sensors, a posture reconstruction method is introduced to estimate the optimal joint positions for consistent motion generation. Based on the high tracking accuracy of the proposed system, we demonstrate that our system is applicable to various motion-based training programs in dance and Taekwondo.

Keywords

References

  1. Vicon Motion Capture System. https://www.vicon.com
  2. OptiTrack Motion Capture System. http://optitrak.com
  3. E. de Aguiar et al., "$M^3$: Marker-Free Model Reconstruction and Motion Tracking from 3D Voxel Data," Conf. Comput. Graph. Applicat., Los Alamitos, CA, USA, Oct. 6-8, 2004, pp. 101-110.
  4. B. Michoud et al., "Real-Time Marker-Free Motion Capture from Multiple Cameras," Int. Conf. Comput. Vis., Rio de Janeiro, Brazil, Oct. 14-21, 2007, pp. 1-7.
  5. S. Corzaaz et al., "A Markerless Motion Capture System to Study Musculoskeletal Biomechanics: Visual Hull and Simulated Annealing Approach," Ann. Biomed. Eng., vol. 34, no. 6, June 2006, pp. 1019-1029. https://doi.org/10.1007/s10439-006-9122-8
  6. E.D. Aguiar et al., "Marker-Less Deformable Mesh Tracking for Human Shape and Motion Capture," IEEE Conf. Comput. Vis. Pattern Recogn., Minneapolis, MN, USA, June 17-22, 2007, pp. 1-8.
  7. J. Gall et al., "Motion Capture Using Joint Skeleton Tracking and Surface Estimation," IEEE Conf. Comput. Vis. Pattern Recogn., Miami, FL, USA, June 20-25, 2009, pp. 1746-1753.
  8. Microsoft Kinect Camera. https://developer.microsoft.com/enus/windows/kinect
  9. L. Chen, H. Wei, and J. Ferryman, "A Survey of Human Motion Analysis Using Depth Imagery," Pattern Recogn. Lett., vol. 23, no. 15, Nov. 2013, pp. 1995-2006.
  10. D.S. Alexiadis et al., "Evaluating a Dancer's Performance Using Kinect-Based Skeleton Tracking," Proc. ACM. Int. Conf. Multimedia, Scottsdale, AZ, USA, Nov. 28-Dec. 1, 2011, pp. 659-662.
  11. S. Izadi et al., "KinectFusion: Real-Time 3D Reconstruction and Interaction Using a Moving Depth Camera," Proc. Annu. ACM Symp. User Interface Softw. Technol., Santa Barbara, CA, USA, Oct. 16-19, 2011, pp. 559-568.
  12. E. Auvinet, J. Meunier, and F. Multon, "Multiple Depth Cameras Calibration and Body Volume Reconstruction for Gait Analysis," Int Conf. Inform. Sci., Signal Process. Their Applicat., Montreal, Canada, July 2-5, 2012, pp. 478-483.
  13. K. Berger et al., Markerless Motion Capture Using Multiple Color-Depth Sensors, Vision, Modeling, and Visualization, Airela Ville, Switzerland: The Eurographics Association, 2011, pp. 317-324.
  14. L. Zhang et al., "Real-Time Human Motion Tracking Using Multiple Depth Cameras," IEEE Int. Conf. Intell. Robots Syst., Vilamoura-Algarve, Portugal, Oct. 7-12, 2012, pp. 2389-2395.
  15. B. Williamson et al., Multi-kinect Tracking for Dismounted Soldier Training, Interservice/Ind. Training, Simulation, Educ. Conf., Orlando, FL, USA, Dec. 3-6, 2012, pp. 1-9.
  16. S. Asteriadis et al., "Estimating Human Motion from Multiple Kinect Sensors," Proc. Int. Conf. Comput. Vis. Comput. Graph.s Collaboration Techn. Applicat., Berlin, Germany, June 6-7, 2013, pp. 3-8.
  17. A. Kitsikidis et al., "Dance Analysis Using Multiple Kinect Sensors," Int. Conf. Comput. Vis. Theory Applicat., Lisbon, Portugal, Jan. 5-8, 2014, pp. 789-795.
  18. S. Kaenchan et al., "Automatic Multiple Kinect Cameras Setting for Simple Walking Posture Analysis," Int. Comput. Sci. Eng. Conf., Bangkok, Thailand, Sept. 4-6, 2013, pp. 245-249.
  19. S. Moon et al., "Multiple Kinect Sensor Fusion for Human Skeleton Tracking Using Kalman Filtering," Int. J. Adv. Robot. Syst., vol. 13, 2016, pp. 1-10. https://doi.org/10.5772/62058
  20. H. Jo et al., "Motion Tracking System for Multi-user with Multiple Kinects," Int. J. u- e-Service, Sci. Technol., vol. 8, no. 7, 2015, pp. 99-108. https://doi.org/10.14257/ijunesst.2015.8.7.10
  21. N. Ahmed, Unified Skeletal Animation Reconstruction with Multiple Kinects, Aire-la Ville, Switzerland: the Eurographics Association, 2014, pp. 5-8.
  22. S. Baek and M. Kim, "Dance Experience System Using Multiple Kinects," Int. J. Future Comput. Commun., vol. 4, no. 1, Feb. 2015, pp. 45-49. https://doi.org/10.7763/IJFCC.2015.V4.353
  23. P.J. Besl and N.D. McKay, "A Method for Registration of 3-D Shapes," Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, Feb. 1992, pp. 239-256. https://doi.org/10.1109/34.121791
  24. W.H. Press et al., Numerical Recipes in C++: the Art of Scientific Computing, New York, USA: Cambridge University Press, 2002.
  25. Xsens MVN Motion Capture System. http://xsens.com
  26. Libgreenect2. https://openkinect.github.io/libfreenect2/
  27. J. Shotton et al., "Real-Time Human Pose Estimation in Parts from Single Depth Images," Commun. ACM, vol. 56, no. 1, Jan. 2013, pp. 116-124. https://doi.org/10.1145/2398356.2398381

Cited by

  1. Development of Dance Learning System Using Human Depth Information vol.18, pp.8, 2017, https://doi.org/10.9728/dcs.2017.18.8.1627
  2. On-Line Detection and Segmentation of Sports Motions Using a Wearable Sensor vol.18, pp.3, 2017, https://doi.org/10.3390/s18030913
  3. Empirical study of the usability and interactivity of an augmented-reality dressing mirror vol.24, pp.10, 2018, https://doi.org/10.1007/s00542-018-3879-1
  4. Dynamic Pose Estimation Using Multiple RGB-D Cameras vol.18, pp.11, 2017, https://doi.org/10.3390/s18113865
  5. Predicting Human Actions Taking into Account Object Affordances vol.93, pp.3, 2017, https://doi.org/10.1007/s10846-018-0815-7
  6. Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning vol.19, pp.7, 2017, https://doi.org/10.3390/s19071716
  7. Monitoring Human Induced Floor Vibrations for Quantifying Dance Moves: A Study of Human-Structure Interaction vol.6, pp.None, 2017, https://doi.org/10.3389/fbuil.2020.00036
  8. Flexible Piezoelectric Nanofibers/Polydimethylsiloxane‐Based Pressure Sensor for Self‐Powered Human Motion Monitoring vol.8, pp.3, 2017, https://doi.org/10.1002/ente.201901242
  9. Multi‐dimensional data modelling of video image action recognition and motion capture in deep learning framework vol.14, pp.7, 2020, https://doi.org/10.1049/iet-ipr.2019.0588
  10. Robust Feature Matching with Spatial Smoothness Constraints vol.12, pp.19, 2020, https://doi.org/10.3390/rs12193158
  11. Optimization of Interactive Animation Capture System for Human Upper Limb Movement Based on XSENS Sensor vol.2021, pp.None, 2021, https://doi.org/10.1155/2021/5246438
  12. Vibrotactile-Based Operational Guidance System for Space Science Experiments vol.10, pp.9, 2017, https://doi.org/10.3390/act10090229
  13. Fusion of Unobtrusive Sensing Solutions for Home-Based Activity Recognition and Classification Using Data Mining Models and Methods vol.11, pp.19, 2021, https://doi.org/10.3390/app11199096
  14. VR-Based Job Training System Using Tangible Interactions vol.21, pp.20, 2017, https://doi.org/10.3390/s21206794