Journal of Software Assessment and Valuation (한국소프트웨어감정평가학회 논문지)

Korea Software Assessment and Valuation Society (한국소프트웨어감정평가학회)
권호 표지
DOI QR코드

DOI QR Code

Divide and Conquer Strategy for CNN Model in Facial Emotion Recognition based on Thermal Images

얼굴 열화상 기반 감정인식을 위한 CNN 학습전략

  • 이동환 (과학기술연합대학원대학교 ICT전공) ;
  • 유장희 (한국전자통신연구원 인공지능연구소)
  • Received : 2021.11.12
  • Accepted : 2021.12.20
  • Published : 2021.12.31
⟨ Previous Next ⟩

Abstract

The ability to recognize human emotions by computer vision is a very important task, with many potential applications. Therefore the demand for emotion recognition using not only RGB images but also thermal images is increasing. Compared to RGB images, thermal images has the advantage of being less affected by lighting conditions but require a more sophisticated recognition method with low-resolution sources. In this paper, we propose a Divide and Conquer-based CNN training strategy to improve the performance of facial thermal image-based emotion recognition. The proposed method first trains to classify difficult-to-classify similar emotion classes into the same class group by confusion matrix analysis and then divides and solves the problem so that the emotion group classified into the same class group is recognized again as actual emotions. In experiments, the proposed method has improved accuracy in all the tests than when recognizing all the presented emotions with a single CNN model.

감정인식은 응용 분야의 다양성으로 많은 연구가 이루어지고 있는 기술이며, RGB 영상은 물론 열화상을 이용한 감정인식의 필요성도 높아지고 있다. 열화상의 경우는 RGB 영상과 비교해 조명 문제에 거의 영향을 받지 않는 장점이 있으나 낮은 해상도로 성능 높은 인식 기술을 필요로 한다. 본 논문에서는 얼굴 열화상 기반 감정인식의 성능을 높이기 위한 Divide and Conquer 기반의 CNN 학습전략을 제안하였다. 제안된 방법은 먼저 분류가 어려운 유사 감정 클래스를 confusion matrix 분석을 통해 동일 클래스 군으로 분류하도록 학습시키고, 다음으로 동일 클래스 군으로 분류된 감정 군을 실제 감정으로 다시 인식하도록 문제를 나누어서 해결하는 방법을 사용하였다. 실험을 통하여, 제안된 학습전략이 제시된 모든 감정을 하나의 CNN 모델에서 인식하는 경우보다 모든 실험에서 높은 인식성능을 보이는 것을 확인하였다.

Keywords

Acknowledgement

이 논문은 2021년도 정부(과학기술정보통신부)의 재원으로 수행된 연구임. (2019-0-00330, 영유아/아동의 발달장애 조기선별을 위한 행동·반응 심리인지 AI 기술 개발)

References

  1. C. M. Tyng, H. U. Amin, M. N. M. Saad, and A. S. Malik, "The Influences of Emotion on Learning and Memory", Frontiers in Psychology, Vol.8, pp.1-22, Aug. 2017. https://doi.org/10.3389/fpsyg.2017.01454
  2. S. Zepf, J. Hernandez, A. Schmitt, W. Minker, and R. W. Picard, "Driver Emotion Recognition for Intelligent Vehicles: A Survey", ACM Computing Surveys, Vol.53, No.3, pp.1-30, June 2020. https://doi.org/10.1145/3388790
  3. E. Yadegaridehkordi, N. F. B. M. Noor, M. N. B. Ayub, H. B. Affal, and N. B. Hussin, "Affective Computing in Education: A Systematic Review and Future Research", Computers & Education, Vol.142, pp.1-19, Dec. 2019. https://doi.org/10.1016/j.compedu.2019.103649
  4. Z. Liu et al., "A Facial Expression Emotion Recognition based Human-Robot Interaction System", in IEEE/CAA Journal of Automatica Sinica, Vol.4, No.4, pp.668-676, Sep. 2017. https://doi.org/10.1109/JAS.2017.7510622
  5. MarketsandMarkets, "Emotion Detection and Recognition Market by Component (Solutions [Facial Expression Recognition, Speech & Voice Recognition] Services), Technology, Application Area, End User, Vertical, Region-Global Forecast to 2026", https://www.marketsandmarkets.com/Market-Reports/emotion-detection-recognition-market-23376176.html, Mar. 2021.
  6. Z. Yu and C. Zhang, "Image based Static Facial Expression Recognition with Multiple Deep Network Learning", in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp.435-442, Seattle, USA, Nov. 2015. https://doi.org/10.1145/2818346.2830595
  7. I. M. Revina and W. R. S. Emmanuel, "A Survey on Human Face Expression Recognition Techniques", Journal of King Saud University-Computer and Information Sciences, Vol.33, No.6, pp.619-628, July 2021. https://doi.org/10.1016/j.jksuci.2018.09.002
  8. K. Zhao, J. Zhao, M. Zhang, Q. Cui, and X. L. Fu, "Neural Responses to Rapid Facial Expressions of Fear and Surprise", Frontiers in Psychology, Vol.8, pp.1-8, May 2017. https://doi.org/10.3389/fpsyg.2017.00761
  9. D. Poster et al., "A Large-scale, Time-synchronized Visible and Thermal Face Dataset", in Proceedings of the 2021 IEEE/CVF Winter Conference on Applications of Computer Vision, pp.1559-1568, Waikoloa, USA, Jan. 2021. https://doi.org/10.1109/WACV48630.2021.00160
  10. C. Ordun, E. Raff and S. Purushotham, "The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data", arXiv preprint arXiv:2009.10589, Sep. 2020. https://arxiv.org/abs/2009.10589
  11. M. Kopaczka, R. Kolk, and D. Merhof, "A Fully Annotated Thermal Face Database and Its Application for Thermal Facial Expression Recognition", in Proceedings of the 2018 IEEE International Instrumentation and Measurement Technology Conference, pp.1-6, Houston, USA, May 2018. https://doi.org/10.1109/I2MTC.2018.8409768
  12. Y. M. Elbarawy, N. I. Ghali, and R. S. El-Sayed, "Facial Expressions Recognition in Thermal Images based on Deep Learning Techniques", International Journal of Image, Graphics and Signal Processing, Vol.11, No.10, pp.1-7, Oct. 2019. https://doi.org/10.5815/ijigsp.2019.10.01
  13. F. He, T. Liu and D. Tao, "Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence", in Advances in Neural Information Processing Systems, pp.1141-1150, Vancouver, Canada, Dec. 2019. https://papers.nips.cc/paper/2019/hash/dc6a70712a252123c40d2adba6a11d84-Abstract.html
  14. H. Wu and X. Gu, "Towards Dropout Training for Convolutional Neural Networks", Neural Networks, Vol.71, pp.1-10, Nov. 2015. https://doi.org/10.1016/j.neunet.2015.07.007
  15. J. M. Johnson and T. M. Khoshgoftaar, "Survey on Deep Learning with Class Imbalance", Journal of Big Data, Vol.6, No.1, pp.1-54, Mar. 2019. https://doi.org/10.1186/s40537-019-0192-5
  16. Besma Abid, "IRIS Thermal/Visible Face Database, IEEE OTCBVS WS Series Bench", vcipl-okstate.org/pbvs/bench/, June 2003.
  17. S. Wang et al., "A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference", in IEEE Transactions on Multimedia, Vol.12, No.7, pp.682-691, Nov. 2010. https://doi.org/10.1109/TMM.2010.2060716
  18. K. Panetta et al., "A Comprehensive Database for Benchmarking Imaging Systems", in IEEE Transactions Pattern Analysis and Machine Intelligence, Vol.42, No.3, pp.509-520, Mar. 2020. https://doi.org/ 10.1109/TPAMI.2018.2884458
  19. J. Deng et al., "Retinaface: Single-stage Dense Face Localisation in the Wild", arXiv preprint arXiv:1905.00641, May 2019. https://arxiv.org/abs/1905.00641
  20. C. Shorten and T. M. Khoshgoftaar, "A Survey on Image Data Augmentation for Deep Learning", Journal of Big Data, Vol.6, No.1, pp.1-48, July 2019. https://doi.org/10.1186/s40537-019-0191-0
  21. K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition", in Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp.770-778, Las Vegas, USA, June 2016. https://doi.org/10.1109/CVPR.2016.90
  22. S. K. M, Kamath, R. Rajendran, Q. Wan, K. Panetta and S. S. Agaian, "TERNet: A Deep Learning Approach for Thermal Face Emotion Recognition", Mobile Multimedia/Image Processing, Security, and Applications 2019, Vol.10993, pp.1-7, May 2019. https://doi.org/10.1117/12.2518708