IEMEK Journal of Embedded Systems and Applications (대한임베디드공학회논문지)

Institute of Embedded Engineering of Korea (대한임베디드공학회)
권호 표지
DOI QR코드

DOI QR Code

Lightweight Deep Learning Model for Heart Rate Estimation from Facial Videos

얼굴 영상 기반의 심박수 추정을 위한 딥러닝 모델의 경량화 기법

  • Received : 2023.01.09
  • Accepted : 2023.03.02
  • Published : 2023.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a deep learning method for estimating the heart rate from facial videos. Our proposed method estimates remote photoplethysmography (rPPG) signals to predict the heart rate. Although there have been proposed several methods for estimating rPPG signals, most previous methods can not be utilized in low-power single board computers due to their computational complexity. To address this problem, we construct a lightweight student model and employ a knowledge distillation technique to reduce the performance degradation of a deeper network model. The teacher model consists of 795k parameters, whereas the student model only contains 24k parameters, and therefore, the inference time was reduced with the factor of 10. By distilling the knowledge of the intermediate feature maps of the teacher model, we improved the accuracy of the student model for estimating the heart rate. Experiments were conducted on the UBFC-rPPG dataset to demonstrate the effectiveness of the proposed method. Moreover, we collected our own dataset to verify the accuracy and processing time of the proposed method on a real-world dataset. Experimental results on a NVIDIA Jetson Nano board demonstrate that our proposed method can infer the heart rate in real time with the mean absolute error of 2.5183 bpm.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea and funded by a grant from the Korean government (No. 2021R1G1A1009792).

References

  1. M. Poh, D. J. McDuff, R. W. Picard, "Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam," IEEE Transactions on Biomedical Engineering, Vol. 58, No. 1, pp. 7-11, 2011.  https://doi.org/10.1109/TBME.2010.2086456
  2. M. Lewandowska, J. Ruminski, T. Kocejko, J. Nowak, "Measuring Pulse Rate with a Webcam - A Non-contact Method for Evaluating Cardiac Activity," Proceedings of 2011 Federated Conference on Computer Science and Information Systems, pp. 405-410, 2011. 
  3. G. de Haan, V. Jeanne, "Robust Pulse Rate From Chrominance-Based rPPG," IEEE Transactions on Biomedical Engineering, Vol. 60, No. 10, pp. 2878-2886, 2013.  https://doi.org/10.1109/TBME.2013.2266196
  4. W. Wang, A. C. den Brinker, S. Stuijk, G. de Haan, "Algorithmic Principles of Remote PPG," IEEE Transactions on Biomedical Engineering, Vol. 64, No. 7, pp. 1479-1491, 2017.  https://doi.org/10.1109/TBME.2016.2609282
  5. W. Chen, D. McDuff, "DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks," Proceedings of the European Conference on Computer Vision, pp. 349-365, 2018. 
  6. Z. Yu, X. Li, G. Zhao, "Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-temporal Networks," Proceedings of the British Machine Vision Conference, 2019. 
  7. E. M. Nowara, D. McDuff, A. Veeraraghavan, "The Benefit of Distraction: Denoising Camera-Based Physiological Measurements Using Inverse Attention," Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4955-4964, 2021. 
  8. Z. Yu, Y. Shen, J. Shi, H. Zhao, P. H. S. Torr, G. Zhao, "PhysFormer: Facial Video-Based Physiological Measurement With Temporal Difference Transformer," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4186-4196, 2022. 
  9. S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, J. Dubois, "Unsupervised Skin Tissue Segmentation for Remote Photoplethysmography," Pattern Recognition Letters, Vol. 124, pp. 82-90, 2019.  https://doi.org/10.1016/j.patrec.2017.10.017
  10. K. Zhang, Z. Zhang, Z. Li, Y. Qiao, "Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks," IEEE Signal Processing Letters, Vol. 23, No. 10, pp. 1499-1503, 2016.  https://doi.org/10.1109/LSP.2016.2603342
  11. G. E. Hinton, O. Vinyals, J. Dean, "Distilling the Knowledge in a Neural Network," ArXiv:1503.02531, 2015. 
  12. S. Lin, H. Xie, B. Wang, K. Yu, X. Chang, X. Liang, G. Wang, "Knowledge Distillation via the Target-Aware Transformer," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10915-10924, 2022. 
  13. E. Lee, E. Chen., Lee, C. Y. Lee, "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner," Proceedings of the European Conference on Computer Vision, pp. 392-409, 2020. 
  14. X. Niu, H. Han, S. Shan, X. Chen, "SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific," Proceedings of the International Conference on Pattern Recognition, pp. 3580-3585, 2018. 
  15. F. Bousefsaf, A. Pruski, C. Maaoui, "3d Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video," Applied Sciences 9, No. 20: 4364, 2019. 
  16. P. Gupta, B. Bhowmick, A. Pal, "Accurate Heart-rate Estimation from Face Videos Using Qualitybased Fusion," Proceedings of the 2017 International Conference on Image Processing, pp. 4132-4136, 2017. 
  17. R. Song, H. Chen, J. Cheng, C. Li, Y. Liu, X. Chen, "PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography," IEEE Journal of Biomedical and Health Informations, Vol. 25. No. 5, pp. 1373-1384, 2021.  https://doi.org/10.1109/JBHI.2021.3051176
  18. Z. Yu, W. Peng, X. Li, X. Hong, G. Zhao, "Remote Heart Rate Measurement from Highly Compressed Facial Videos: An End-to-end Deep Learning Solution with Video Enhancement," Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 151-160, 2019.