Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Emotion prediction neural network to understand how emotion is predicted by using heart rate variability measurements

  • Park, Sung Soo (SKKU Business School, Sungkyunkwan University) ;
  • Lee, Kun Chang (SKKU Business School/SAIHST (Samsung Advanced Institute for Health Sciences & Technology)/Healthcare Informatics Research Center, Sungkyunkwan University)
  • Received : 2017.02.23
  • Accepted : 2017.03.23
  • Published : 2017.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

Correct prediction of emotion is essential for developing advanced health devices. For this purpose, neural network has been successfully used. However, interpretation of how a certain emotion is predicted through the emotion prediction neural network is very tough. When interpreting mechanism about how emotion is predicted by using the emotion prediction neural network can be developed, such mechanism can be effectively embedded into highly advanced health-care devices. In this sense, this study proposes a novel approach to interpreting how the emotion prediction neural network yields emotion. Our proposed mechanism is based on HRV (heart rate variability) measurements, which is based on calculating physiological data out of ECG (electrocardiogram) measurements. Experiment dataset with 23 qualified participants were used to obtain the seven HRV measurement such as Mean RR, SDNN, RMSSD, VLF, LF, HF, LF/HF. Then emotion prediction neural network was modelled by using the HRV dataset. By applying the proposed mechanism, a set of explicit mathematical functions could be derived, which are clearly and explicitly interpretable. The proposed mechanism was compared with conventional neural network to show validity.

Keywords

References

  1. D. E. Bloom, E. T. Cafiero, E. Jane-Llopis, S. Abrahams-Gessel, L. R. Bloom, S. Fathima, A. B. Feigl, T. Gaziano, M. Mowafi, A. Pandya, K. Prettner, L. Rosenberg, B. Seligman, A. Z. Stein, and C. Weinstein, "The Global Economic Burden of Non-Communicable Diseases," Geneva, Switzerland: World Economic Forum, July 2011.
  2. J. Mervis, "Agencies rally to tackle big data," Science, Vol. 336, No.6077, pp. 22-22, 2012. https://doi.org/10.1126/science.336.6077.22
  3. K. R. Kim, J. G. Yun, and S. G. Yoon, "Analysis on the Status of Korean Major Firms Entering into HT Fusion New Business," Korea Health Industry Development Institute, 2014. Download https://www.khidi.or.kr /fileDownload?titleId=128633&fileId=1&fileDownType =C¶mMenuId=MENU01101
  4. J. A. Sacristan, "Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients," BMC Medical Informatics and Decision Making, Vol. 13, No. 1, pp. 6-13, 2013. https://doi.org/10.1186/1472-6947-13-6
  5. M. Phelan, L. Stradins, and S. Morrison, (2001). "Towards a global definition of patient centred care," British Medical Journal, Vol. 322, pp. 444-445, 2001. https://doi.org/10.1136/bmj.322.7284.444
  6. M. Ozkaynak, P. F. Brennan, D. A. Hanauer, S. Johnson, J. Aarts, K. Zheng, and S. N. Haque, "Patient-centered care requires a patientoriented workflow model," Journal of the American Medical Informatics Association, Vol. 20, No. e1, pp. e14-e16, 2013. https://doi.org/10.1136/amiajnl-2013-001633
  7. C. M. Tucker, T. M. Arthur, J. Roncoroni, W. Wall, and J. Sanchez, "Patient-centered, culturally sensitive health care," American Journal of Lifestyle Medicine, Vol. 9, No. 1, pp. 63-77, 2015. https://doi.org/10.1177/1559827613498065
  8. S. M. Yoon and M. H. Lee, "Current Status and Prospects of Life Appliance based Healthcare System," The Journal of The Korean Institute of Communication Sciences, Vol. 31, No. 12, pp. 31-37, Nov. 2014.
  9. Y. L. Zheng, X. R. Ding, C. C. Y. Poon, B. P. L. Lo, H. Zhang, X. L. Zhou, ... and Y. T. Zhang, "Unobtrusive sensing and wearable devices for health informatics," IEEE Transactions on Biomedical Engineering, Vol. 61, No. 5, pp. 1538-1554, 2014. https://doi.org/10.1109/TBME.2014.2309951
  10. A. J. Wearden, N. Tarrier, C. Barrowclough, T. R. Zastowny, and A. A. Rahill, "A review of expressed emotion research in health care," Clinical psychology review, Vol. 20, No. 5, pp. 633-666, 2000. https://doi.org/10.1016/S0272-7358(99)00008-2
  11. B. Freshman, and L. Rubino, "Emotional intelligence: a core competency for health care administrators," The health care manager, Vol. 20, No. 4, pp. 1-9, 2002. https://doi.org/10.1097/00126450-200206000-00002
  12. Y. F. Birks, and I. S. Watt, "Emotional intelligence and patient-centred care," Journal of the Royal Society of Medicine, Vol. 100, No. 8, pp. 368-374, 2007. https://doi.org/10.1177/014107680710000813
  13. S. Mann, "A health-care model of emotional labour: an evaluation of the literature and development of a model," Journal of health organization and management, Vol. 19, No. 4/5, pp. 304-317, 2005. https://doi.org/10.1108/14777260510615369
  14. C. D. Katsis, N. S. Katertsidis, and D. I. Fotiadis, "An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders," Biomedical Signal Processing and Control, Vol. 6, No. 3, pp. 261-268, 2011. https://doi.org/10.1016/j.bspc.2010.12.001
  15. P. A. Kragel, and K. S. LaBar, "Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions," Emotion, Vol. 13, No. 4, pp. 681-690, 2013. https://doi.org/10.1037/a0031820
  16. J. Selvaraj, M. Murugappan, K. Wan, and S. Yaacob, "Classification of emotional states from electrocardiogram signals: a non-linear approach based on hurst," Biomedical engineering online, Vol. 12, No. 44, 2013.
  17. G. Valenza, L. Citi, A. Lanata, E. P. Scilingo, and R. Barbieri, "Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics," Scientific reports, Vol. 4, No. 4998, pp. 1-13, 2014.
  18. M. Swangnetr, and D. B. Kaber, "Emotional state classification in patient-robot interaction using wavelet analysis and statistics-based feature selection," IEEE Transactions on Human-Machine Systems, Vol. 43, No. 1, pp. 63-75, 2013. https://doi.org/10.1109/TSMCA.2012.2210408
  19. D. Kukolja, S. Popovic, M. Horvat, B. Kovac, and K. Cosic, "Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications," International Journal of Human-Computer Studies, Vol. 72, No. 10, pp. 717-727, 2014. https://doi.org/10.1016/j.ijhcs.2014.05.006
  20. C. M. Bishop, "Neural Networks for Pattern Recognition" Clarendon Press, Oxford, 1995.
  21. F. Stahl and I. Jordanov, "An overview of use of neural networks for data mining tasks," WIREs: Data Mining and Knowledge Discovery, pp. 193-208, 2012.
  22. J. A. Anderson, "An Introduction to Neural Networks" MIT press, 1995.
  23. B. D. Ripley, "Pattern Recognition and Neural Networks" Cambridge University Press, 1996.
  24. G. D. Garson, "Interpreting neural-network connection weights," AI Expert, Vol. 6, pp. 47-51, 1991.
  25. L. K. Milne, "Feature selection with neural networks with contribution measures," In Proceedings of the Australian Conference on Artificial Intelligence AI'95, Canberra, 1995.
  26. J. M. Zurada, A. Malinowski, and I. Cloete, "Sensitivity analysis for minimization of input data dimension for feedforward neural network," Proceedings of IEEE International Symposium on Circuits and Systems, Vol. 6, pp. 447-450, May 1994.
  27. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning Internal Representations by Error Propagation," Parallel Distributed Processing, Vol. 1, pp. 318-362, 1986.
  28. S. Haykin, "Neural Network" Prentice Hall, 1994.
  29. P. De Wilde, "Neural network models: theory and projects" Springer Science & Business Media, 2013.
  30. A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion recognition using bio-sensors: First steps towards an automatic system," In Tutorial and research workshop on affective dialogue systems, Springer Berlin Heidelberg, pp. 36-48, June 2004.
  31. L. Ma, and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 34, No. 3, pp. 1588-1595, 2004. https://doi.org/10.1109/TSMCB.2004.825930
  32. M. M. Monwar, and S. Rezaei, "Pain recognition using artificial neural network. In Signal Processing and Information Technology," 2006 IEEE International Symposium on Signal Processing and Information Technology, pp. 28-33, August 2006.
  33. Y. S. Seol, D. J. Kim, and H. W. Kim, "Emotion recognition from text using knowledge-based ANN," 23rd International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), pp. 1569-1572, July 2008.
  34. C. N. Moridis, and A. A. Economides, "Prediction of student's mood during an online test using formula-based and neural network-based method," Computers & Education, Vol. 53, No. 3, pp. 644-652, 2009. https://doi.org/10.1016/j.compedu.2009.04.002
  35. M. A. Nicolaou, H. Gunes, and M. Pantic, "Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space," IEEE Transactions on Affective Computing, Vol. 2, No. 2, pp. 92-105, 2011. https://doi.org/10.1109/T-AFFC.2011.9
  36. F. A. Russo, N. N. Vempala, and G. M. Sandstrom, "Predicting musically induced emotions from physiological inputs: Linear and neural network models," Frontiers in Psychology, Vol. 4, No. 468, 2013.
  37. M. P. Tarvainen, J. P. Niskanen, J. A. Lipponen, P. O. Ranta-Aho, and P. A. Karjalainen, "Kubios HRV-heart rate variability analysis software," Computer methods and programs in biomedicine, Vol. 113, No. 1, pp. 210-220, 2014. https://doi.org/10.1016/j.cmpb.2013.07.024
  38. R. Hecht-Nielsen, "Theory of the Back Propagation Neural Network," Proceeding of the International Joint Conference on Neural Networks, pp. 593-605, 1989.
  39. I. H. Witten, and E. Frank, "Data Mining: Practical machine learning tools and techniques" Morgan Kaufmann, 2005.
  40. S. W. Byun, S. P. Lee, "Emotion Recognition Using Tone and Tempo Based on Voice for IoT," The transactions of The Korean Institute of Electrical Engineers, Vol. 65, No. 1, pp. 116-121, 2016. https://doi.org/10.5370/KIEE.2016.65.1.116
  41. Y. J. Kang, "Smart Affect Jewelry based on Multi-modal," Journal of the Korea Institute of Information and Communication Engineering, Vol. 20, No. 7, pp. 1317-1324, 2016. https://doi.org/10.6109/jkiice.2016.20.7.1317
  42. P. S. Kim, "Technology and development trends related to human-friendly emotional robots," The Journal of The Korean Institute of Communication Sciences, 33(8), 19-27, 2016.
  43. C. D. Katsis, N. S. Katertsidis, and D. I. Fotiadis, "An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders," Biomedical Signal Processing and Control, Vol. 6, No. 3, pp. 261-268, 2011. https://doi.org/10.1016/j.bspc.2010.12.001