Science of Emotion and Sensibility (감성과학)

Korean Society for Emotion and Sensibility (한국감성과학회)
권호 표지
DOI QR코드

DOI QR Code

Affective Representation of Behavioral and Physiological Responses to Emotional Videos using Wearable Devices

웨어러블 기구를 이용한 영상 자극에 대한 행동 및 생리적 정서 표상

  • Received : 2023.06.08
  • Accepted : 2023.07.19
  • Published : 2024.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

This study examined affective representation by analyzing physiological responses measured using wearable devices and affective ratings in response to emotional videos. To achieve this aim, a published dataset was reanalyzed using multidimensional scaling to demonstrate affective representation in two dimensions. Cross-participant classification was also conducted to identify the consistency of emotional responses across participants. The accuracy and misclassification in each emotional condition were described by exploring the confusion matrix derived from the classification analysis. Multidimensional scaling revealed that the represented objects, namely, emotional videos, were positioned along the rated valence and arousal vectors, supporting the core affect theory (Russell, 1980). Vector fittings of physiological responses also showed the associations between heart rate acceleration and low arousal, increased heart rate variability and negative and high arousal, and increased electrodermal activity and negative and low arousal. Using the data of behavioral and physiological responses across participants, the classification results revealed that emotional videos were more accurately classified than the chance level of classification. The confusion matrix showed that awe, enthusiasm, and liking, which were categorized as positive, low arousal emotions in this study, were less accurately classified than the other emotions and were misclassified for each other. Through multivariate analyses, this study confirms the core affect theory using physiological responses measured through wearable devices and affective ratings in response to emotional videos.

본 연구는 정서 영상을 보며 웨어러블 기구로 측정된 생리적 반응과 정서평정을 분석하여 유발된 정서가 어떻게 표상되는지 알아보고자 하였다. 연구 목적을 위해, 공유된 데이터셋을 다차원척도법(multidimensional scaling)을 통해 정서 영상, 생리적 반응 및 정서 평정을 2차원에 표상하였다. 또한, 참가자간 분류분석(cross-participant classification)을 활용해 참가자 간 정서표상이 얼마나 일관적인지 분석하였다. 추가적으로, 참가자들의 반응이 유사한 정도가 각 정서 조건 별로 다른지 확인하기 위해, 정서 영상 별 정확분류와 오분류를 혼동행렬(confusion matrix)을 통해 탐색하였다. 다차원척도법 결과, 정서 영상들과 정서 평정의 위치가 기존 이론과 부합하게 정서가 및 각성가 벡터에 따라 표상되어, Russell(1980)의 핵심정서이론을 지지하는 것을 확인했다. 표상된 생리적 반응 벡터를 통해, 심박률 증가-저각성, 높은 심박률변산성-부정정서 고각성, 피부전기활동 증가-부정정서 저각성의 관계를 시각화했다. 행동 및 생리 데이터로 학습한 참가자간 분류분석 결과, 평균 정확도가 우연수준보다 높았다. 이는 동일한 영상에 대한 참가자들의 공유되는 정서 표상이 있음을 지지한다. 혼동행렬표를 통해, 저각성 긍정정서로 표상된 감탄, 열정, 그리고 선호는 상대적으로 잘 분류되지 않았고, 서로 더 많이 오분류되는 것을 확인하였다. 다변량 분석인 다차원척도법과 분류분석을 통해, 본 연구는 영상 자극에 웨어러블 기구로 측정한 생리적 반응과 정서 평정도 핵심 정서 이론과 부합하는 결과를 얻은 것에 의의가 있다.

Keywords

Acknowledgement

이 논문은 한국연구재단 4단계 BK21사업(전북대학교 심리학과)의 지원을 받아 연구되었음(No.4199990714213).

References

  1. Bach, D. R., Friston, K. J., & Dolan, R. J. (2010). Analytic measures for quantification of arousal from spontaneous skin conductance fluctuations. International Journal of Psychophysiology, 76(1), 52-55. DOI: 10.1016/J.IJPSYCHO.2010.01.011
  2. Baucom, L. B., Wedell, D. H., Wang, J., Blitzer, D. N., & Shinkareva, S. V. (2012). Decoding the neural representation of affective states. Neuroimage, 59(1), 718-727. DOI: 10.1016/j.neuroimage.2011.07.037
  3. Eom, J., Park, H., Noh, J., & Sohn, J.-H. (2011). Cardiovascular response to surprise stimulus. Korean Society for Emotion and Sensibility, 14(1), 147-156.
  4. Gross, J. J., & Levenson, R. W. (1995). Emotion elicitation using films. Cognition & Emotion, 9(1), 87-108.
  5. Golland, Y., Keissar, K., & Levit-Binnun, N. (2014). Studying the dynamics of autonomic activity during emotional experience. Psychophysiology, 51(11), 1101-1111. DOI:10.1111/psyp.12261
  6. Golland, Y., Hakim, A., Aloni, T., Schaefer, S., & Levit-Binnun, N. (2018). Affect dynamics of facial EMG during continuous emotional experiences. Biological Psychology, 139, 47-58. DOI: 10.1016/j.biopsycho.2018.10.003
  7. Jang, E., Eum, Y., Park, B., Kim, S., & Sohn, J. (2011). Difference of autonomic nervous system responses among boredom, pain, and surprise. Science of Emotion and Sensibility, 14(4), 503-512.
  8. Jang, J., & Kim, J. (2023). Consistency between individuals of affective responses for multiple modalities based on behavioral and physiological data. Science of Emotion and Sensibility, 26(1), 43-54. DOI: 10.14695/kjsos.2023.26.1.43
  9. Jo, A., Kim, H., & Kim, J. (2023). Affective representation and consistency across individuals responses to affective videos. Science of Emotion and Sensibility, 26(3), 15-28. DOI: 10.14695/KJSOS.2023.26.3.15
  10. Kim, H., & Kim, J. (2022). Affective responses to asmr using multidimensional scaling and classification. Korean Society for Emotion and Sensibility, 25(3), 47-62. DOI: 10.14695/kjsos.2022. 25.3.47
  11. Kim, J., & Wedell, D. H. (2016). Comparison of physiological responses to affect eliciting pictures and music. International Journal of Psychophysiology, 101, 9-17. DOI: 10.1016/j.ijpsycho.2015.12.011
  12. Kim, J., Weber, C. E., Gao, C., Schulteis, S., Wedell, D. H., & Shinkareva, S. V. (2020). A study in affect: Predicting valence from fMRI data. Neuropsychologia, 143, 107473. DOI: 10.1016/j.neuropsychologia.2020.107473
  13. Kazmi, S. Z., Zhang, H., Aziz, W., Monfredi, O., Abbas, S. A., Shah, S. A., Kazmi, S. S., & Butt, W. H. (2016). Inverse correlation between heart rate variability and heart rate demonstrated by linear and nonlinear analysis. PLoS One, 11(6), e0157557. DOI: 10.1371/journal.pone.0157557.
  14. Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: A review. Biological Psychology, 84(3), 394-421.
  15. Kreibig, S. D., Wilhelm, F. H., Roth, W. T., & Gross, J. J. (2007). Cardiovascular, electrodermal, and respiratory response patterns to fear-and sadness- inducing films. Psychophysiology, 44(5), 787-806. DOI: 10.1111/j.1469-8986.2007.00550.x
  16. Park, C., Kim, I., & Kim. J. (2023). Affective representations of basic tastes and intensity using multivariate analyses. Korean Society for Emotion and Sensibility, 26(2), 39-52. DOI: 10.14695/KJSOS.2023.26.2.39
  17. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161. DOI:10.1037/h0077714
  18. Saganowski, S., Komoszynska, J., Behnke, M., Perz, B., Kunc, D., Klich, B., Kaczmarek, L. D., & Kazienko, P. (2022). Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables. Scientific Data, 9(1), 158. DOI: 10.1038/s41597-022-01262-0
  19. Soleymani, M., Villaro-Dixon, F., Pun, T., & Chanel, G. (2017). Toolbox for emotional feAture extraction from physiological signals (TEAP). Frontiers in ICT, 4. 1. DOI: 10.3389/fict.2017.00001/full
  20. Stuldreher, I. V., van Erp, J. B., & Brouwer, A. M. (2023). Robustness of physiological synchrony in wearable electrodermal activity and heart rate as a measure of attentional engagement to movie clips. Sensors, 23(6), 3006. DOI: 10.3390/s23063006