Journal of Information Processing Systems

  • 제17권6호
  • /
  • Pages.1138-1156
  • /
  • 2021
  • /
  • 1976-913X(pISSN)
  • /
  • 2092-805X(eISSN)
한국정보처리학회 (Korea Information Processing Society)
권호 표지
DOI QR코드

DOI QR Code

A Survey on Image Emotion Recognition

  • Zhao, Guangzhe (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture) ;
  • Yang, Hanting (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture) ;
  • Tu, Bing (School of Information Science and Technology, Hunan Institute of Science and Technology) ;
  • Zhang, Lei (College of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture)
  • 투고 : 2019.10.11
  • 심사 : 2020.05.14
  • 발행 : 2021.12.31
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

Emotional semantics are the highest level of semantics that can be extracted from an image. Constructing a system that can automatically recognize the emotional semantics from images will be significant for marketing, smart healthcare, and deep human-computer interaction. To understand the direction of image emotion recognition as well as the general research methods, we summarize the current development trends and shed light on potential future research. The primary contributions of this paper are as follows. We investigate the color, texture, shape and contour features used for emotional semantics extraction. We establish two models that map images into emotional space and introduce in detail the various processes in the image emotional semantic recognition framework. We also discuss important datasets and useful applications in the field such as garment image and image retrieval. We conclude with a brief discussion about future research trends.

키워드

과제정보

This work was supported by the Pyramid Talent Training Project of BUCEA and funded by open research fund program of Beijing Key Laboratory of Robot Bionics and Function Research.

참고문헌

  1. W. Wang and Q. He, "Emotion-based image semantic query through color description," Journal of South China University of Technology (Natural Science Edition), vol. 36, no. 1, pp. 60-66, 2008. https://doi.org/10.3321/j.issn:1000-565X.2008.01.011
  2. T. C. Merlo, I. Soletti, E. Saldana, B. S. Menegali, M. M. Martins, A. C. B. Teixeira, S. D. S. Harada-Padermo, M. D. B. Dargelio, and C. J. Contreras-Castillo, "Measuring dynamics of emotions evoked by the packaging colour of hamburgers using Temporal Dominance of Emotions (TDE)," Food Research International, vol. 124, pp. 147-155, 2019. https://doi.org/10.1016/j.foodres.2018.08.007
  3. J. Yi, A. Chen, Z. Cai, Y. Sima, M. Zhou, and X. Wu, "Facial expression recognition of intercepted video sequences based on feature point movement trend and feature block texture variation," Applied Soft Computing, vol. 82, article no. 105540, 2019. https://doi.org/10.1016/j.asoc.2019.105540
  4. A. Barman and P. Dutta, "Facial expression recognition using distance and texture signature relevant features," Applied Soft Computing, vol. 77, pp. 88-105, 2019. https://doi.org/10.1016/j.asoc.2019.01.011
  5. S. M. Alarcao, "Reminiscence therapy improvement using emotional information," in Proceedings of 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 561-565.
  6. S. Wang, K. Han, and J. Jin, "Review of image low-level feature extraction methods for content-based image retrieval," Sensor Review, vol. 39, no. 6, pp. 783-809, 2019. https://doi.org/10.1108/sr-04-2019-0092
  7. J. Yao, Y. Yu, and X. Xue, "Sentiment prediction in scene images via convolutional neural networks," in Proceedings of 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC), 2 Wuhan, China, 016, pp. 196-200.
  8. C. Colombo, A. Del Bimbo, and P. Pala, "Semantics in visual information retrieval," IEEE Multimedia, vol. 6, no. 3, pp. 38-53, 1999. https://doi.org/10.1109/93.790610
  9. N. Jamil, S. Lqbal, and N. Iqbal, "Face recognition using neural networks," in Proceedings of IEEE International Multi Topic Conference: Technology for the 21st Century (INMIC), Lahore, Pakistan, 2001, pp. 277-281.
  10. Y. Yu, Z. Tian, and Z. Cai, "Research of perceptive information of image," Acta Electronica Sinica, vol. 29, no. 10, pp. 1373-1375, 2001. https://doi.org/10.3321/j.issn:0372-2112.2001.10.019
  11. Z. Li, Z. Tan, L. Cao, H. Chen, L. Jiao, and Y. Zhong, "Directive local color transfer based on dynamic lookup table," Signal Processing: Image Communication, vol. 79, pp. 1-12, 2019. https://doi.org/10.1016/S0165-1684(99)00076-6
  12. H. Zhang, T. Huang, and L. Liu, "Study on the emotion factor space of fabric," in Proceedings of the International Conference on Kansei Engineering and Emotion Research, Penghu, Taiwan, 2012, pp. 1174-1178.
  13. T. Hayashi and M. Hagiwara, "Image query by impression words-the IQI system," IEEE Transactions on Consumer Electronics, vol. 44, no. 2, pp. 347-352, 1998. https://doi.org/10.1109/30.681949
  14. S. Wang, E. Chen, J. Li, and X. Wang, "Kansei-based image evaluation and retrieval," Pattern Recognition and Artificial Intelligence, vol. 14, no. 3, pp. 297-301, 2001. https://doi.org/10.3969/j.issn.1003-6059.2001.03.009
  15. S. Zhao, X. Zhao, G. Ding, and K. Keutzer, "EmotionGAN: unsupervised domain adaptation for learning discrete probability distributions of image emotions," in Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea, 2018, pp. 1319-1327.
  16. H. Sadeghi and A. A. Raie, "Human vision inspired feature extraction for facial expression recognition," Multimedia Tools and Applications, vol. 78, no. 21, pp. 30335-30353, 2019. https://doi.org/10.1007/s11042-019-07863-z
  17. S. Zhao, H. Yao, Y. Gao, R. Ji, and G. Ding, "Continuous probability distribution prediction of image emotions via multitask shared sparse regression," IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 632-645, 2016. https://doi.org/10.1109/TMM.2016.2617741
  18. L. Cai, H. Xu, Y. Yang, and J. Yu, "Robust facial expression recognition using RGB-D images and multichannel features," Multimedia Tools and Applications, vol. 78, no. 20, pp. 28591-28607, 2019. https://doi.org/10.1007/s11042-018-5981-x
  19. F. Liu, B. Wang, and Q. Zhang, "Deep learning of pre-classification for fast image retrieval," in Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence, Sanya, China, 2018, pp. 1-5.
  20. K. Stallman, Emotional Psychology. Liaoning, China: Liaoning People Press, 1986.
  21. R. Plutchik, "The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice," American Scientist, vol. 89, no. 4, pp. 344-350, 2001. https://doi.org/10.1511/2001.4.344
  22. M. Y. Tsalamlal, M. A. Amorim, J. C. Martin, and M. Ammi, "Combining facial expression and touch for perceiving emotional valence," IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 437-449, 2016. https://doi.org/10.1109/taffc.2016.2631469
  23. Y. Y. Gao, X. P. Wang, and Y. X. Yin, "Research on affective annotation for natural scene images," Journal of Chinese Computer Systems, vol. 32, no. 4, pp. 767-771, 2011.
  24. Y. Ye, X. Zhang, Y. Lin, and H. Wang, "Facial expression recognition via region-based convolutional fusion network," Journal of Visual Communication and Image Representation, vol. 62, pp. 1-11, 2019. https://doi.org/10.1016/j.jvcir.2019.04.009
  25. Q. Dai and Y. Yu, "A kind of image retrieval based on texture features and BP neural network," Computer Science, vol. 27, no. 6, pp. 55-57, 2000. https://doi.org/10.3969/j.issn.1002-137X.2000.06.015
  26. A. R. Kurup, M. Ajith, and M. M. Ramon, "Semi-supervised facial expression recognition using reduced spatial features and deep belief networks," Neurocomputing, vol. 367, pp. 188-197, 2019. https://doi.org/10.1016/j.neucom.2019.08.029
  27. J. Lee and E. Park, "Fuzzy similarity-based emotional classification of color images," IEEE Transactions on Multimedia, vol. 13, no. 5, pp. 1031-1039, 2011. https://doi.org/10.1109/TMM.2011.2158530
  28. R. V. Priya, "Emotion recognition from geometric fuzzy membership functions," Multimedia Tools and Applications, vol. 78, no. 13, pp. 17847-17878, 2019. https://doi.org/10.1007/s11042-018-6954-9
  29. D. Palani and K. Venkatalakshmi, "An IoT based predictive modelling for predicting lung cancer using fuzzy cluster based segmentation and classification," Journal of Medical Systems, vol. 43, article no. 21, 2019. https://doi.org/10.1007/s10916-018-1139-7
  30. C. Singh, E. Walia, and K. P. Kaur, "Enhancing color image retrieval performance with feature fusion and non-linear support vector machine classifier," Optik, vol. 158, pp. 127-141, 2018. https://doi.org/10.1016/j.ijleo.2017.11.202
  31. Z. Sun, G. Bebis, and R. Miller, "Quantized wavelet features and support vector machines for on-road vehicle detection," in Proceedings of 7th International Conference on Control, Automation, Robotics and Vision, Singapore, 2002, pp. 1641-1646.
  32. S. Zhao, Y. Gao, X. Jiang, H. Yao, T. S. Chua, and X. Sun, "Exploring principles-of-art features for image emotion recognition," in Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, 2014, pp. 47-56.
  33. Q. You, J. Luo, H. Jin, and J. Yang, "Building a large scale dataset for image emotion recognition: the fine print and the benchmark," in Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, 2016, pp. 308-314.
  34. S. Corchs, E. Fersini, and F. Gasparini, "Ensemble learning on visual and textual data for social image emotion classification," International Journal of Machine Learning and Cybernetics, vol. 10, no. 8, pp. 2057-2070, 2019. https://doi.org/10.1007/s13042-017-0734-0
  35. L. Poretski, J. Lanir, and O. Arazy, "Feel the image: the role of emotions in the image-seeking process," Human-Computer Interaction, vol. 34, no. 3, pp. 240-277, 2019. https://doi.org/10.1080/07370024.2017.1359604
  36. A. Hernandez-Garcia, "Perceived emotion from images through deep neural networks," in Proceedings of 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 566-570.
  37. D. Kong, X. Shen, L. Cao, and G. Jin, "Phase retrieval for attacking fractional Fourier transform encryption," Applied Optics, vol. 56, no. 12, pp. 3449-3456, 2017. https://doi.org/10.1364/AO.56.003449
  38. S. Wang, E. Chen, J. Li, and X. Wang, "Kansei-based image evaluation and retrieval," Pattern Recognition and Artificial Intelligence, vol. 14, no. 3, pp. 297-301, 2001 https://doi.org/10.3969/j.issn.1003-6059.2001.03.009
  39. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, UK: Cambridge University Press, 2004.
  40. C. C. Chang and C. J. Lin, "LIBSVM: a library for support vector machines," ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, article no. 27, 2011. https://doi.org/10.1145/1961189. 1961199
  41. S. Wang, E. Chen, Z. Wang, and Z. Wang, "Research of emotion semantic image annotation and retrieval algorithm using support vector machine," Pattern Recognition and Artificial Intelligence, vol. 17, no. 1, pp. 27-33, 2004. https://doi.org/10.3969/j.issn.1003-6059.2004.01.006