DOI QR코드

DOI QR Code

Facial Data Visualization for Improved Deep Learning Based Emotion Recognition

  • Lee, Seung Ho
  • Received : 2019.03.05
  • Accepted : 2019.05.29
  • Published : 2019.06.30

Abstract

A convolutional neural network (CNN) has been widely used in facial expression recognition (FER) because it can automatically learn discriminative appearance features from an expression image. To make full use of its discriminating capability, this paper suggests a simple but effective method for CNN based FER. Specifically, instead of an original expression image that contains facial appearance only, the expression image with facial geometry visualization is used as input to CNN. In this way, geometric and appearance features could be simultaneously learned, making CNN more discriminative for FER. A simple CNN extension is also presented in this paper, aiming to utilize geometric expression change derived from an expression image sequence. Experimental results on two public datasets (CK+ and MMI) show that CNN using facial geometry visualization clearly outperforms the conventional CNN using facial appearance only.

Keywords

facial expression recognition;convolutional neural network;facial landmark points;facial geometry visualization

References

  1. Yang, J., Zhang, D. D., Frangi, A. F., & Yang, J. Y. (2004). Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), 131-137. https://doi.org/10.1109/TPAMI.2004.1261097
  2. Zafeiriou, S., & Petrou, M. (2010). Sparse representation for facial expressions recognition via l1 optimization. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13-18 June (pp. 32-39). Piscataway: IEEE.
  3. Meng, Z., Liu, P., Cai, J., Han, S., & Tong, Y. (2017). Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, USA, 30 May-3 June (pp. 558-565). Piscataway: IEEE.
  4. Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005). Web-based database for facial expression analysis. In 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, Netherlands, 6 July (p. 5). Piscataway: IEEE.
  5. Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012). Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing, 30(10), 683-697. https://doi.org/10.1016/j.imavis.2012.06.005
  6. Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137-154. https://doi.org/10.1023/B:VISI.0000013087.49260.fb
  7. Wang, Z., & Ying, Z. (2012). Facial expression recognition based on local phase quantization and sparse representation. In 2012 8th International Conference on Natural Computation, Chongqing, China, 29-31 May (pp. 222-225). Piscataway: IEEE.
  8. Weng, Z., Wang, S., & Ji, Q. (2013). Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23-28 June (pp. 3422-3429).
  9. Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 210-227.
  10. Yang, H., Ciftci, U., & Yin, L. (2018). Facial expression recognition by de-expression residue learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-23 June (pp. 2168-2177).
  11. Donato, G., Stewart, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 974-989. https://doi.org/10.1109/34.799905
  12. Huang, M. W., Wang, Z. W., & Ying, Z. L. (2010). A new method for facial expression recognition based on sparse representation plus LBP. In 2010 3rd International Congress on Image and Signal Processing, Yantai, China, 16-18 October (Vol. 4, pp. 1750-1754). Piscataway: IEEE.
  13. Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015). Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7-13 December (pp. 2983-2991).
  14. Kotsia, I., & Pitas, I. (2007). Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing, 16(1), 172-187. https://doi.org/10.1109/TIP.2006.884954
  15. Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. https://doi.org/10.1109/5.726791
  16. Lee, S. H., Kim, H., Ro, Y. M., & Plataniotis, K. N. (2013). Using color texture sparsity for facial expression recognition. In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China, 22-26 April (pp. 1-6). Piscataway: IEEE.
  17. Lee, S. H., & Ro, Y. M. (2016). Partial matching of facial expression sequence using over-complete transition dictionary for emotion recognition. IEEE Transactions on Affective Computing, 7(4), 389-408. https://doi.org/10.1109/TAFFC.2015.2496320
  18. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13-18 June (pp. 94-101). Piscataway: IEEE.
  19. Asthana, A., Zafeiriou, S., Cheng, S., & Pantic, M. (2014). Incremental face alignment in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23-28 June (pp. 1859-1866).
  20. Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2005). Recognizing facial expression: Machine learning and application to spontaneous behavior. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20-25 June (pp. 568-573). Piscataway: IEEE.
  21. Chen, J., Chen, D., Gong, Y., Yu, M., Zhang, K., & Wang, L. (2012). Facial expression recognition using geometric and appearance features. In Proceedings of the 4th International Conference Internet Multimedia Computing and Service, Wuhan, China (pp. 29-33). New York: ACM.
  22. Choi, J. Y., Neve, W. D., Plataniotis, K. N., & Ro, Y. M. (2011). Collaborative face recognition for improved face annotation in personal photo collections shared on online social networks. IEEE Transactions on Multimedia, 13(1), 14-28. https://doi.org/10.1109/TMM.2010.2087320

Acknowledgement

Supported by : KOREATECH