Science of Emotion and Sensibility (감성과학)

Korean Society for Emotion and Sensibility (한국감성과학회)
권호 표지
DOI QR코드

DOI QR Code

Emotion Recognition Method Using Heart-Respiration Connectivity

심장과 호흡의 연결성을 이용한 감성인식 방법

  • Lee, Dong Won (Department of Emotion Engineering, Sangmyung University) ;
  • Park, Sangin (Industry-Academy Cooperation Foundation, Sangmyung University) ;
  • Whang, Mincheol (Department of Intelligent Engineering Informations for Human, Sangmyung University)
  • 이동원 (상명대학교 감성공학과) ;
  • 박상인 (상명대학교 산학협력단) ;
  • 황민철 (상명대학교 미래융합공학대학 휴먼지능정보공학부)
  • Received : 2016.09.08
  • Accepted : 2017.06.15
  • Published : 2017.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Physiological responses have been measured to recognize emotion. Although physiological responses have been interrelated between organs, their connectivities have been less considered for emotion recognizing. The connectivities have been assumed to enhance emotion recognition. Specially, autonomic nervous system is physiologically modulated by the interrelated functioning. Therefore, this study has been tried to analyze connectivities between heart and respiration and to find the significantly connected variables for emotion recognition. The eighteen subjects(10 male, age $24.72{\pm}2.47$) participated in the experiment. The participants were asked to listen to predetermined sound stimuli (arousal, relaxation, negative, positive) for evoking emotion. The bio-signals of heart and respiration were measured according to sound stimuli. HRV (heart rate variability) and BRV (breathing rate variability) spectrum were obtained from spectrum analysis of ECG (electrocardiogram) and RSP (respiration). The synchronization of HRV and BRV spectrum was analyzed according to each emotion. Statistical significance of relationship between them was tested by one-way ANOVA. There were significant relation of synchronization between HRV and BRV spectrum (synchronization of HF: F(3, 68) = 3.605, p = 0.018, ${\eta}^2_p=0.1372$, synchronization of LF: F(3, 68) = 5.075, p = 0.003, ${\eta}^2_p=0.1823$). HF difference of synchronization between ECG and RSP has been able to classify arousal from relaxation (p = 0.008, d = 1.4274) and LF's has negative from positive (p = 0.002, d = 1.7377). Therefore, it was confirmed that the heart and respiration to recognize the dimensional emotion by connectivity.

감성을 인식하는데 있어 생리적 반응은 중요하다. 생리적 반응은 인체의 주요 기관들과 밀접한 관련이 있지만 감성을 인식하는데 연결성은 고려되지 않고 있다. 자율신경계는 감성과 밀접한 관련이 있는데, 심장과 폐와 같은 인체 내 주요 내장기관에 분포되어 기능적 상보작용을 통해 생리적 반응을 조절하기 때문이다. 따라서 본 연구는 심장과 호흡의 연결성을 분석하고 감성을 인식하는 중요한 연결 변수를 찾고자 하였다. 피험자 18명(남 10명, 평균 나이 $24.72{\pm}2.47$)은 소리 자극을 이용한 감성 유발 실험에 참여하였고 심전도와 호흡 데이터를 측정하였다. 수집된 심장과 호흡 데이터는 스펙트럼 분석을 이용하여 HRV와 BRV spectrum을 구하였고, 감성에 따른 HRV와 BRV spectrum의 동기화 차이를 일원배치분산분석을 통해 통계적 유의성을 확인하였다. Tukey 검증 결과, arousal-relaxation은 HF 대역에서 심전도와 호흡의 동기화 차이로 인식 가능하였고(p = 0.008, d = 1.4274), negative-positive는 LF 대역에서 인식이 가능하였다(p = 0.002, d = 1.7377). 본 연구 결과로 심장과 호흡의 연결성을 통해 차원적 감성을 정량적으로 평가할 수 있음을 확인하였고, 복합적인 원인으로 발현되는 감성을 인식하는데 생리적 반응들의 연결성 변수의 활용도가 높을 것으로 기대된다.

Keywords

References

  1. Ahn, S., Whang, M., Kim, D. K., Kim, J., & Park, S. (2012). Real-time emotion recognition technology using individualization process emotional technology. Science of Emotion & Sensibility, 15(1), 133-140.
  2. Andreassi, J. L. (2013). Psychophysiology: Human Behavior & Physiological Response. Psychology Press.
  3. Castellano, G., Kessous, L., & Caridakis, G. (2008). Emotion recognition through multiple modalities: face, body gesture, speech. Affect and Emotion in Human-Computer Interaction, 92-103. DOI: 10.1007/978-3-540-85099-1
  4. Cowie, R., & Douglas-Cowie, E. (1996). Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In Spoken Language Fourth International Conference, 3, 1989-1992. DOI: 10.1109/ICSLP.1996.608027
  5. Darwin, C. (1998). The expression of the emotions in man and animals. USA: Oxford University Press.
  6. Ekman, P., & Friesen, W. V. (1977). Facial action coding system: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto.
  7. Glass, L., & Shrier, A. (1991). Low-dimensional dynamics in the heart. In Theory of Heart, 289-312. DOI:10.1007/978-1-4612-3118-9_12
  8. Gross, J. J., Fredrickson, B. L., & Levenson, R. W. (1994). The psychophysiology of crying. Psychophysiology, 31(5), 460-468. doi: 10.1111/j.1469-8986.1994.tb01049.x
  9. Gunes, H., & Piccardi, M. (2007). Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications, 30(4), 1334-1345. DOI: 10.1016/j.jnca.2006.09.007
  10. Hirsch, J. A., & Bishop, B. (1981). Respiratory sinus arrhythmia in humans: how breathing pattern modulates heart rate. American Journal of Physiology- Heart and Circulatory Physiology, 241(4), 620-629. https://doi.org/10.1152/ajpheart.1981.241.4.H620
  11. Ioannou, S. V., Raouzaiou, A. T., Tzouvaras, V. A., Mailis, T. P., Karpouzis, K. C., & Kollias, S. D. (2005). Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Networks, 18(4), 423-435. DOI: 10.1016/j.neunet.2005.03.004
  12. Kamath, M. V., & Fallen, E. L. (1992). Power spectral analysis of heart rate variability: a noninvasive signature of cardiac autonomic function. Critical Reviews in Biomedical Engineering, 21(3), 245-311.
  13. Kang, S. A., & Cho, S. (2013). Economic impact of emotional ICT industry on korean economy using input-output analysis (감성 ICT 산업의 경제적 파급효과 분석). Korean Industrial Economic Association, 26(4), 1739-1764.
  14. Katona, P. G., & Jih, F. (1975). Respiratory sinus arrhythmia: noninvasive measure of parasympathetic cardiac control. Journal of Applied Physiology, 39(5), 801-805. https://doi.org/10.1152/jappl.1975.39.5.801
  15. Khalfa, S., Roy, M., Rainville, P., Dalla Bella, S., & Peretz, I. (2008). Role of tempo entrainment in psychophysiological differentiation of happy and sad music?. International Journal of Psychophysiology, 68(1), 17-26. DOI: 10.1016/j.ijpsycho.2007.12.001
  16. Kwon, O. W., Chan, K., Hao, J., & Lee, T. W. (2003). Emotion recognition by speech signals. In Eighth European Conference on Speech Communication and Technology.
  17. Lee, Y. J., & Youn, I. (2015). Emotion recognition technology for Human-machine interface (인간-기계 인터페이스를 위한 감성인식 기술). The Korean Society of Mechanical Engineers, 55(3), 42-46.
  18. Levenson, R. W. (1992). Autonomic nervous system differences among emotions. Psychological Science, 3(1), 23-27. DOI: 10.1111/j.1467-9280.1992.tb00251.x
  19. Lieberman, P., & Michaels, S. B. (1962). Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech. The Journal of the Acoustical Society of America, 34(7), 922-927. DOI: 10.1121/1.1918222
  20. Malliani, A., Pagani, M., Lombardi, F., & Cerutti, S. (1991). Cardiovascular neural regulation explored in the frequency domain. Circulation, 84(2), 482-492. DOI: 10.1161/01.CIR.84.2.482
  21. McCraty, R., Atkinson, M., Tomasino, D., & Bradley, R. T. (2009). The coherent heart: heart-brain interactions, psychophysiological coherence, and the emergence of system-wide order. Integral Review, 5(2), 10-115.
  22. Nyklicek, I., Thayer, J. F., & Van Doornen, L. J. (1997). Cardiorespiratory differentiation of musicallyinduced emotions. Journal of Psychophysiology, 11(4), 304-321.
  23. Porges, S. W. (1995). Orienting in a defensive world: Mammalian modifications of our evolutionary heritage. A polyvagal theory. Psychophysiology, 32(4), 301-318. DOI: 10.1111/j.1469-8986.1995.tb01213.x
  24. Prkachin, K. M., Williams-Avery, R. M., Zwaal, C., & Mills, D. E. (1999). Cardiovascular changes during induced emotion: An application of Lang's theory of emotional imagery. Journal of Psychosomatic Research, 47(3), 255-267. DOI: 10.1016/S0022-3999(99)00036-7
  25. Rainville, P., Bechara, A., Naqvi, N., & Damasio, A. R. (2006). Basic emotions are associated with distinct patterns of cardiorespiratory activity. International journal of psychophysiology, 61(1), 5-18. DOI: 10.1016/j.ijpsycho.2005.10.024
  26. Rottenberg, J., Wilhelm, F. H., Gross, J. J., & Gotlib, I. H. (2003). Vagal rebound during resolution of tearful crying among depressed and nondepressed individuals. Psychophysiology, 40(1), 1-6. DOI:10.1111/1469-8986.00001
  27. Saul, J. P. (1990). Beat-to-beat variations of heart rate reflect modulation of cardiac autonomic outflow. Physiology, 5(1), 32-37. https://doi.org/10.1152/physiologyonline.1990.5.1.32
  28. Tiller, W. A., McCraty, R., & Atkinson, M. (1996). Cardiac coherence: A new, noninvasive measure of autonomic nervous system order. Alternative Therapies in Health and Medicine, 2(1), 52-65.
  29. Yasuma, F., & Hayano, J. I. (2004). Respiratory sinus arrhythmia: why does the heartbeat synchronize with respiratory rhythm?. Chest Journal, 125(2), 683-690. DOI: 10.1378/chest.125.2.683
  30. Yoon, H. S., Han, Y. J., & Hahn, H. S. (2009). Real-time recognition system of facial expressions using principal component of Gabor-wavelet features. Journal of Korean Institute of Intelligent Systems, 19(6), 821-827. DOI: 10.5391/JKIIS.2009.19.6.821
  31. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39-58. DOI: 10.1109/TPAMI.2008.52