Journal of the Institute of Convergence Signal Processing (융합신호처리학회논문지)

The Korea Institute of Convergence Signal Processing (한국융합신호처리학회)
권호 표지

Emotion Recognition based on Multiple Modalities

  • Kim, Dong-Ju (Department of Electrical and Computer Engineering from Sungkyunkwan University) ;
  • Lee, Hyeon-Gu (Department of Information and Communication Engineering at Seoil University) ;
  • Hong, Kwang-Seok (Department of Electrical and Computer Engineering from Sungkyunkwan University)
  • Received : 2011.05.14
  • Accepted : 2011.11.01
  • Published : 2011.10.30
Download PDF
⟨ Previous Next ⟩

Abstract

Emotion recognition plays an important role in the research area of human-computer interaction, and it allows a more natural and more human-like communication between humans and computer. Most of previous work on emotion recognition focused on extracting emotions from face, speech or EEG information separately. Therefore, a novel approach is presented in this paper, including face, speech and EEG, to recognize the human emotion. The individual matching scores obtained from face, speech, and EEG are combined using a weighted-summation operation, and the fused-score is utilized to classify the human emotion. In the experiment results, the proposed approach gives an improvement of more than 18.64% when compared to the most successful unimodal approach, and also provides better performance compared to approaches integrating two modalities each other. From these results, we confirmed that the proposed approach achieved a significant performance improvement and the proposed method was very effective.

Keywords

References

  1. Picard, R., "Affective computing", MIT Press, Boston, 1997.
  2. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G., "Emotion recognition in human-computer interaction". IEEE Signal Processing Magazine, 2001.
  3. M. Song, J. Bu, C. Chen, N. Li, "Audio-visual based emotion recognition-a new approach", Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1020-1025, 2004.
  4. Chang Ryu, Min An, Yoon Na, Jin Cho, Young Han, Kyoung Kim and Pyung Park, "A portable neurofeedback system and EEG-analysis methods for evaluation", World Congress on Medical Physics and Biomedical Engineering 2006, vol. 14, no. 8, pp. 1167-1169, 2007.
  5. Schaaff, K., Schultz, T., "Towards an EEG-based emotion recognizer for humanoid robots", The 18th IEEE International Symposium on Robot and Human Interactive Communication, pp. 792-796, 2009.
  6. M.Turk, A. Pentland, "Eigenfaces for recognition", J. Cognitive Neuro-sci,, vol. 3, no. 1, pp. 71-86, 1991. https://doi.org/10.1162/jocn.1991.3.1.71
  7. Y. Jian, Z. David, F. Alejandro, and J. Y. Yang, "Two-dimensional PCA: A new approach to appearance-based face representation and recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 131-137, 2004. https://doi.org/10.1109/TPAMI.2004.1261097
  8. D. O'Shaughnessy, "Speech Communication - Human and Machine", Addison-Wesley, New York, 1987.
  9. Reynolds, D. A, Quatieri, T., Dunn, R., "Speaker verification using adapted Gaussian mixture models", Digital Signal Process, vol. 10, pp. 19-41, 2000. https://doi.org/10.1006/dspr.1999.0361
  10. P. Jourlin, J. Luettin, D. Genond, H. Wassner, "Acoustic-labial speaker authentication", Pattern Recognition Letter. vol. 18. pp. 853-858, 1997. https://doi.org/10.1016/S0167-8655(97)00070-6
  11. A. Ross, A.K. Jain, "Information fusion in biometrics", Pattern Recognition Letter, vol. 24, pp. 2115-2125, 2003. https://doi.org/10.1016/S0167-8655(03)00079-5
  12. Michael J. Lyons, Shigeru Akamatsu, Miyuki Kamachi & Jiro Gyoba, "Coding Facial Expressions with Gabor Wavelets", Proceedings, Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200-205, 1998.
  13. Kyung Hak Hyun, Eun Ho Kim, Yoon Keun Kwak, "Improvement of emotion recognition by Bayesian classifier using non-zero-pitch concept", IEEE International Workshop on Robot and Human Interactive Communication, pp. 312-316, 2005.
  14. P. Lang, M. Bradley, and B. Cuthbert. "International affective picture system (lAPS): Affective ratings of pictures and instruction manual", Technical Report A-6, University of Florida, Gainesville, 2005.
  15. Papanicolaou A. C, Loring D. W, Deutsch G, Eisenberg H. M, "Task-related EEG Asymmetries: a Comparison of Alpha Blocking and Beta Enhancement", International Journal of Neuroscience, vol. 30, pp. 81-85, 1998.