A Multimodal Emotion Recognition Using the Facial Image and Speech Signal

  • Go, Hyoun-Joo (Chungbuk National University, School of Electrical and Computer Engineering Research Institute for computer and Information Communication) ;
  • Kim, Yong-Tae (Hankyong National University Department of Information and Control Engineering) ;
  • Chun, Myung-Geun (Chungbuk National University, School of Electrical and Computer Engineering Research Institute for computer and Information Communication)
  • Published : 2005.03.01


In this paper, we propose an emotion recognition method using the facial images and speech signals. Six basic emotions including happiness, sadness, anger, surprise, fear and dislike are investigated. Facia] expression recognition is performed by using the multi-resolution analysis based on the discrete wavelet. Here, we obtain the feature vectors through the ICA(Independent Component Analysis). On the other hand, the emotion recognition from the speech signal method has a structure of performing the recognition algorithm independently for each wavelet subband and the final recognition is obtained from the multi-decision making scheme. After merging the facial and speech emotion recognition results, we obtained better performance than previous ones.


  1. J. Lien, T. Kanade, C. Li, 'Detection, tracking, and classification of action units in facial expression', Journal of Robotics and Autonomous Systems, Vol. 31, No.3, pp. 131-146, 2000
  2. M. Turk, A. Pentland, 'Eigenfaces for recognition' Journal of Cognitive Neuroscience, Vol. 3, No.1, pp.71-86, 1991
  3. P. Penev, J. Atick, 'Local feature analysis: a general statistical theory for object representation', Network : Computation in Neural Systems, Vol. 7, pp. 477-500, 1996
  4. P. Belhumeur, J. Hespanha, D. Kriegman, 'Eigenfaces vs. fisherfaces: Recognition using class specific linear projection', IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 19, No.7, pp. 711-720, 1997
  5. Brown, G. and Satoshi, Y. and Luebben, H. and Sejnowski, T.J., Separation of optically recorded action potential trains in tritonia by ICA, In: Proc. 5th Annual Joint Symposium on Neural Computation, 1998
  6. Marian Stewart Bartlett, Javier R. Movellan, 'Face Recognition by Independent component Analysis' IEEE transactions on neural networks, Vol 13
  7. V.Kostov and S.Fukuda, Emotion in User Interface, Voice Interaction System, IEEE Inti Conf. on Systems, Man, Cybernetics Representation, no. 2, pp. 798-803, 2000
  8. T. Moriyama and S. Oazwa, Emotion Recognition and Synthisis System on Speech IEEE IntI. Conference on Multimedea Computing and Systems, pages 840-844, 1999
  9. L.C. Silva and P.C. Ng, Bimodal Emotion Recognition, Proceeding of the 4th International Conference on Automatic Face and Gesture Recognition, pp. 332-335, 2000
  10. P.Ekman and W.V. Friesen. Emotion in the human face System. Cambridge University Press, San Francisco, CA, second edition, 1982
  11. Burrus, Gopinath, Guo, 'Introduction to Wavelets and Wavelet Transforms A Primer' Prentice-Hall International, Inc, 1998
  12. Gianluca Donato, Marian Stewart Bartlett, Joseph C. Hager 'Classifying Facial Actions' IEEE transactions on pattern analysis and machine intelligence, Vol. 21, No 10, 1999
  13. Roger Jang, Chuen-Tsai Sun, Neuro-fuzzy and Soft computing. Prentice-Hall International, 1997

Cited by

  1. Multimodal Biometric Recognition System using Real Fuzzy Vault vol.23, pp.4, 2013,