KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Detecting Complex 3D Human Motions with Body Model Low-Rank Representation for Real-Time Smart Activity Monitoring System

  • Jalal, Ahmad (Department of Computer Science & Engineering, Air University) ;
  • Kamal, Shaharyar (School of Computer Engineering, Mid-Sweden University) ;
  • Kim, Dong-Seong (School of Electronic Engineering, Kumoh National Institute of Technology)
  • Received : 2017.06.18
  • Accepted : 2017.10.17
  • Published : 2018.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

Detecting and capturing 3D human structures from the intensity-based image sequences is an inherently arguable problem, which attracted attention of several researchers especially in real-time activity recognition (Real-AR). These Real-AR systems have been significantly enhanced by using depth intensity sensors that gives maximum information, in spite of the fact that conventional Real-AR systems are using RGB video sensors. This study proposed a depth-based routine-logging Real-AR system to identify the daily human activity routines and to make these surroundings an intelligent living space. Our real-time routine-logging Real-AR system is categorized into two categories. The data collection with the use of a depth camera, feature extraction based on joint information and training/recognition of each activity. In-addition, the recognition mechanism locates, and pinpoints the learned activities and induces routine-logs. The evaluation applied on the depth datasets (self-annotated and MSRAction3D datasets) demonstrated that proposed system can achieve better recognition rates and robust as compare to state-of-the-art methods. Our Real-AR should be feasibly accessible and permanently used in behavior monitoring applications, humanoid-robot systems and e-medical therapy systems.

Keywords

References

  1. S. Khawandi, B. Daya, and P. Chauvet, "Integrated monitoring system for fall detection in elderly," in Proc. of IEEE Conference on Audio, Language and Image Processing (ICALIP), pp. 62-67, July 16-18, 2012.
  2. X. Zhang, T. Huang, Y. Tian, and W. Gao, "Background-modeling-based adaptive prediction for surveillance video coding," IEEE Transactions on Image Processing, vol. 23, no. 2, pp. 769-784, Feburary, 2013. https://doi.org/10.1109/TIP.2013.2294549
  3. A. Jalal and M. A. Zeb, "Security enhancement for e-learning portal," International Journal of Computer Science and Network Security, vol. 8, pp. 41-45, 2008.
  4. J. Jo, S. Lee, and J. W. Kim, "Software-defined home networking devices for multi-home visual sharing," IEEE Transactions on Consumer Electronics, vol. 60, no. 3, pp. 534-539, March, 2014. https://doi.org/10.1109/TCE.2014.6937340
  5. A. Jalal and S. Kim, "The Mechanism of Edge Detection using the Block Matching Criteria for the Motion Estimation," Human Computer Interaction, pp.484-489, January, 2005.
  6. A. Jalal and I. Uddin, "Security architecture for third generation 3G using GMHS cellular network," in Proc. of IEEE Conference on in Emerging Technologies, pp. 74-79, November 12-13, 2007.
  7. Y. He and Y. Li, "Physical activity recognition utilizing the built-in kinematic sensors of a smartphone," International Journal of Distributed Sensor Networks, vol. 2013, pp. 1-10, 2013. https://doi.org/10.1155/2013/469076
  8. A. Jalal and Y. Rasheed, "Collaboration achievement along with performance maintenance in video streaming," in Proc. of IEEE Conference on Interactive Computer Aided Learning, pp. 1-8, September 26-28, 2007.
  9. A. Jalal, MZ. Uddin, J. T. Kim, and T. Kim, "Daily Human Activity Recognition Using Depth Silhouettes and R Transformation for Smart Home," in Proc. of International Conference on Smart Homes and Health Telematics, pp. 25-32, 20-22 June, 2011.
  10. A. Jalal, S. Kamal and D. Kim, "Facial expression recognition using 1D transform features and Hidden Markov model," Journal of Electrical Engineering and Technology, vol. 12, no. 4, pp. 1657-1662, 2017. https://doi.org/10.5370/JEET.2017.12.4.1657
  11. E. Kim, S. Helal and D. Cook, "Human activity recognition and pattern discovery," IEEE Pervasive Computing, vol. 9, no. 1, pp. 48-53, January, 2010. https://doi.org/10.1109/MPRV.2010.7
  12. A. Jalal, M. Z. Uddin, and T.-S. Kim, "Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home," IEEE Transactions on Consumer Electronics, vol. 58, no. 3, pp. 863-871, March, 2012. https://doi.org/10.1109/TCE.2012.6311329
  13. A. Jalal and S. Kim, "Algorithmic implementation and efficiency maintenance of real-time environment using low-bitrate wireless communication," in Proc. of Proceedings IEEE International workshop on software technologies for future embedded and ubiquitous systems, 2006.
  14. A. Jalal, S. Kamal, and D. Kim, "A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments," Sensors, vol. 14, no. 7, pp. 11735-11759, July, 2014. https://doi.org/10.3390/s140711735
  15. A. Farooq, A. Jalal and S. Kamal, "Dense RGB-D Map-Based Human Tracking and Activity Recognition using Skin Joints Features and Self-Organizing Map," KSII Transactions on Internet and Information Systems, vol. 9, no. 5, pp. 1856-1869, May, 2015. https://doi.org/10.3837/tiis.2015.05.017
  16. A. Jalal, "IM-DailyDepthActivity dataset," Intelligent Media Lab, 2017, [Online; accessed 14 April- 2017].
  17. A. Jalal, M. Z. Uddin, J. T. Kim, and T.-S. Kim, "Recognition of human home activities via depth silhouettes and R transformation for smart homes," Indoor and Built Environment, vol. 21, no. 1, pp. 184-190, September, 2011. https://doi.org/10.1177/1420326X11423163
  18. A. Jalal, J. T. Kim, and T.-S. Kim, "Human activity recognition using the labeled depth body parts information of depth silhouettes," International Symposium on Sustainable Healthy Buildings, pp. 1-8, 27-29 February, 2012.
  19. W. Lin, M.-T. Sun, R. Poovandran, and Z. Zhang, "Human activity recognition for video surveillance," in Proc. of IEEE International Symposium on Circuits and Systems, pp. 2737-2740, 18-21 May, 2008.
  20. A. Jalal, Y. Kim, S. Kamal, A. Farooq and D. Kim, "Human daily activity recognition with joints plus body features representation using Kinect sensor," in Proc. of IEEE Conference on Informatics, Electronics and Vision, pp. 1-6, 15-18 June, 2015.
  21. A. Jalal and Y. Kim, "Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data," in Proc. of IEEE Conference on Advanced Video and Signal-based Surveillance, pp. 119-124, 26-29 August, 2014.
  22. A. Jalal, S. Kamal, and D. Kim, "Shape and motion features approach for activity tracking and recognition from Kinect video camera," in Proc. of IEEE Conference on Advanced Information Networking and Applications, pp. 445-450, 24-27 March, 2015.
  23. L. Xia, C.-C. Chen, and J. K. Aggarwal, "View invariant human action recognition using histograms of 3D joints," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 20-27, 16-21 June, 2012.
  24. O. Oreifej and Z. Liu, "HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 716-723, 23-28 June, 2013.
  25. S. Kamal, A. Jalal and D. Kim, "Depth Images-based Human Detection, Tracking and Activity Recognition Using Spatiotemporal Features and Modified HMM," Journal of Electrical Engineering and Technology, pp. 1921-1926, 2016.
  26. Y. Ming, G. Wang and C. Fan, "Uniform Local Binary Pattern Based Texture-Edge Feature for 3D Human Behavior Recognition," Plos One, vol. 10, no. 5, pp. 1-15, 2015.
  27. K. Charalampous and A. Gasteratos, "On-line deep learning method for action recognition," Pattern analysis and aplications, vol. 19, no. 2, pp. 337-354, 2016. https://doi.org/10.1007/s10044-014-0404-8
  28. A. Jalal and S. Kim, "Global security using human face understanding under vision ubiquitous architecture system," World Academy of Science, Engineering, and Technology, vol. 13, pp. 7-11, 2006.
  29. L. Piyathilaka and S. Kodagoda, "Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features," in Proc. of IEEE Conference on Industrial Electronics and Applications, pp. 567-572, 19-21 June, 2013.
  30. A. Jalal, Y. Kim, and D. Kim, "Ridge body parts features for human pose estimation and recognition from RGB-D video data," in Proc. of IEEE Conference on Computing, Communication and Networking Technologies, pp. 1-6, 11-13 July, 2014.
  31. N.-G. Cho, Y.-J. Kim, U. Park, J.-S. Park and S.-W. Lee, "Group activity recognition with group interaction zone based on relative distance between human objects," International Journal of Pattern Recognition and Artifical intelligence, vol. 29, no. 5, pp. 1-15, August, 2015.
  32. A. Jalal and M. A. Zeb, "Security and QoS Optimization for distributed real time environment," in Proc. of IEEE Conference on Computer and Information Technology, pp. 369-374, 16-19 October, 2007.
  33. A. Jalal and A. Shahzad, "Multiple facial feature detection using vertex modeling structure," in Proc. of IEEE Conference on Interactive Computer Aided Learning, pp. 1-7, 26-28 September, 2007.
  34. A. Jalal, S. Lee, J. Kim, and T. Kim, "Human activity recognition via the features of labeled depth body parts," in Proceedings Smart Homes Health Telematics, pp. 246-249, 2012.
  35. A. Jalal, J. T. Kim, and T.-S Kim, "Development of a life logging system via depth imaging-based human activity recognition for smart homes," in Proc. of Proceedings of the International Symposium on Sustainable Healthy Buildings, pp. 91-95, 2012.
  36. A. Jalal, S. Kamal and D. Kim, "Individual Detection-Tracking-Recognition using depth activity images," in Proc. of IEEE Conference on Ubiquitous Robots and Ambient Intelligence, pp. 450-455, 28-30 October, 2015.
  37. A. Jalal, S. Kim and B. J. Yun, "Assembled algorithm in the real-time H.263 codec for advanced performance," in Proc. of IEEE Workshop on Enterprise Networking and Computing in Healthcare Industry, pp.295-298, 23-25 June, 2005.
  38. A. Jalal, and S. Kim, "A complexity removal in the floating point and rate control phenomenon," Korea Multimedia Society, pp. 48-51, 2005.
  39. A. Jalal and S. Kim, S, "Advanced performance achievement using multi-algorithmic approach of video transcoder for low bit rate wireless communication," International Journal on Graphics, Vision and Image Processing, vol. 5, no. 9, pp. 27-32, September 2005.
  40. M. Rossi, S. Benatti, E. Farella and L. Benini, "Hybrid EMG classifier based on HMM and SVM for hand gesture recognition in prosthetics," in Proc. of IEEE Conference on Industrial Technology, pp. 1700-1705, 17-19 March, 2015.
  41. A. Jalal, N. Sharif, J. T. Kim, and T.-S. Kim, "Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart homes," Indoor and Built Environment, vol. 22, pp. 271-279, 2013. https://doi.org/10.1177/1420326X12469714
  42. S. Kamal and A. Jalal, "A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors," Arabian Journal for Science and Engineering, vol. 41, no. 3, pp. 1043-1051, March 2016. https://doi.org/10.1007/s13369-015-1955-8
  43. A. Jalal, S. Kamal, and D. Kim, "Depth map-based human activity tracking and recognition Using body joints features and self-organized map," in Proc. of IEEE Conference on Computing, Communication and Networking Technologies, pp. 1-6, 11-13 July, 2014.
  44. A. Jalal, S. Kamal and D. Kim, "Depth Silhouettes Context: A new robust feature for human tracking and activity recognition based on embedded HMMs," in Proc. of IEEE 12th International Conference on Ubiquitous Robots and Ambient Intelligence, pp. 294-299, 28-30 October, 2015.
  45. A. Jalal, S. Kamal, A. Farooq and D. Kim, "A spatiotemporal motion variation features extraction approach for human tracking and pose-based action recognition," in Proc. of IEEE Conference on Informatics, Electronics and Vision, pp. 1-6, 15-18 June, 2015.
  46. T. Kim, A. Jalal, H. Han, H. Jeon and J. Kim, "Real-Time Life Logging via Depth Imaging-based Human Activity Recognition towards Smart Homes Services," in Proc. of Proceedings International Symposium on Renewable energy sources and healthy buildings, pp. 63, 2013.
  47. A. Jalal, Y.-H. Kim, Y.-J. Kim, S. Kamal and D. Kim, "Robust human activity recognition from depth video using spatiotemporal multi-fused features," Pattern Recognition, vol. 61, no. 1, pp. 295-308, January 2017. https://doi.org/10.1016/j.patcog.2016.08.003
  48. A. Kundu, Y. He and M.-Y. Chen, "Alternatives to variable duration HMM in handwriting recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1275-1280, November 1998. https://doi.org/10.1109/34.730561
  49. A. Jalal, Shaharyar Kamal and Daijin Kim, "Human depth sensors-based activity recognition using spatiotemporal features and hidden markov model for smart environments," Journal of computer networks and communications, vol. 2016, pp. 1-11, 2016.
  50. A. Jalal and S. Kamal, "Real-Time Life Logging via a Depth Silhouette-based Human Activity Recognition System for Smart Home Services," in Proc. of Proceedings of the IEEE International Conference on Advanced Video and Signal-based Surveillance, pp. 74-80, 2014.
  51. J. M. Carmona and J. Climent, "Action recognition using Rf transform on Optical flow images," in Proc. of International Joint Conference on Computer vision, imaging and computer graphics theory and Applications, pp. 266-271, 27 February, 2017.
  52. D.-M. Tsai and W.-Y. Chiu, "A real-time ICA-based activity recognition in video sequences," in Proc. of International Conference on Machine vision Applications, pp. 467-470, 20-23 May, 2013.
  53. W. Li, Z. Zhang and Z. Liu, "Action recognition based on a bag of 3d points," in Proc. of Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 9-14, 13-18 June, 2010.
  54. X. Yang and Y. Tian, "Eigenjoints-based action recognition using naive-bayes-nearest-neighbor," in Proc. of Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 14-19, 16-21 June, 2012.
  55. A. Vieira, E. Nascimentio, G. Oliveira, Z. Liu, M. Campos, Stop: Space-time occupancy patterns for 3d action recognition from depth map sequences, in Proc. of Proceedings of the Iberoamerican Congress on Pattern Recognition, pp. 252-259, 3-6 September, 2012.

Cited by

  1. Head Pose Detection for a Wearable Parrot-Inspired Robot Based on Deep Learning vol.8, pp.7, 2018, https://doi.org/10.3390/app8071081
  2. Exploring Environmental Factors Affecting Strawberry Yield Using Pattern Recognition Techniques vol.20, pp.1, 2018, https://doi.org/10.7472/jksii.2019.20.1.39
  3. A Study of Accelerometer and Gyroscope Measurements in Physical Life-Log Activities Detection Systems vol.20, pp.22, 2018, https://doi.org/10.3390/s20226670
  4. HF-SPHR: Hybrid Features for Sustainable Physical Healthcare Pattern Recognition Using Deep Belief Networks vol.13, pp.4, 2018, https://doi.org/10.3390/su13041699
  5. Stochastic Remote Sensing Event Classification over Adaptive Posture Estimation via Multifused Data and Deep Belief Network vol.13, pp.5, 2021, https://doi.org/10.3390/rs13050912