Convolutional Neural Network Model Using Data Augmentation for Emotion AI-based Recommendation Systems

Abstract

In this study, we propose a novel research framework for the recommendation system that can estimate the user's emotional state and reflect it in the recommendation process by applying deep learning techniques and emotion AI (artificial intelligence). To this end, we build an emotion classification model that classifies each of the seven emotions of angry, disgust, fear, happy, sad, surprise, and neutral, respectively, and propose a model that can reflect this result in the recommendation process. However, in the general emotion classification data, the difference in distribution ratio between each label is large, so it may be difficult to expect generalized classification results. In this study, since the number of emotion data such as disgust in emotion image data is often insufficient, correction is made through augmentation. Lastly, we propose a method to reflect the emotion prediction model based on data through image augmentation in the recommendation systems.

본 연구에서는 딥러닝 기법과 정서적 AI를 적용하여 사용자의 감정 상태를 추정하고 이를 추천 과정에 반영할 수 있는 추천 시스템에 대한 새로운 연구 프레임워크를 제안한다. 이를 위해 분노, 혐오, 공포, 행복, 슬픔, 놀람, 중립의 7가지 감정을 각각 분류하는 감정분류모델을 구축하고, 이 결과를 추천 과정에 반영할 수 있는 모형을 제안한다. 그러나 일반적인 감정 분류 데이터에서는 각 레이블 간 분포 비율의 차이가 크기 때문에 일반화된 분류 결과를 기대하기 어려울 수 있다. 본 연구에서는 감정 이미지 데이터에서 혐오감 등의 감정 개수가 부족한 경우가 많으므로 데이터 증강을 이용한다. 마지막으로, 이미지 증강을 통해 데이터 기반의 감정 예측 모델을 추천시스템에 반영하는 방법을 제안한다.

I. Introduction

A recommendation system is an indispensable element to enhance the competitiveness of a company in the information age where a lot of information is mass-produced. Many previous studies have proposed novel recommendation systems and techniques. In addition, many studies are being conducted to consider various data that can be acquired in a big data environment to the recommendation process. In particular, as high-dimensional unstructured data such as multimedia, video, and audio as well as unstructured data such as text becomes abundant, research to reflect this information in the recommendation process is newly proposed. Through this, one of the traditional problems of recommendation systems, the sparsity of the user-product matrix, can be supplemented because the preference score, which is missing in the user-product matrix, can be compensated to some extent by using other information. However, in the case of unstructured data, there are many high-dimensional data with severe noise, so it is often difficult to reflect it in the recommendation process. Therefore, it can be said that an appropriate preprocessing process is required to reflect these data in the recommendation process.

Recently, many studies using deep learning have been proposed for the processing of large-volume, high-dimensional, and unstructured data, and the results are actually very good. This is because traditional machine learning generally shows excellent performance in processing structured data, but processing large amounts of unstructured data requires many parameters and cannot be properly processed without using deep learning. In fact, several prior studies suggest ways to utilize new data that can compensate for the sparsity problem of recommendation systems using deep learning.

In this study, we propose a method to reflect the user's facial expression information among various unstructured data to improve the performance of the recommendation system. Facial expression information can estimate human emotional state by converting human facial expressions into data. A technology capable of estimating such human emotional states through data such as images and voices is called emotion AI (artificial intelligence) or affective computing. Traditionally, it has been difficult to collect image data or voice data such as facial expression data, and it has been very difficult to estimate human emotional states through them. However, in recent years, interest in emotion AI has increased as it has become possible to precisely estimate using a convolutional neural network or a deep neural network.

In this study, we propose a novel research framework for the recommendation system that can estimate the user's emotional state and reflect it in the recommendation process by applying deep learning techniques and emotion AI. Usually, a user purchases a product while looking at the screen of a monitor or mobile device. In this process, when a product recommendation is made, the user's emotional state is highly likely to appear in the facial expression. Recently, cameras are installed in most mobile devices, and if the user agrees, the user's facial expression data captured through the camera can be collected. As it reflects the emotional state of the person, it will be possible to make more sophisticated recommendations.

To this end, we first develop an emotion AI system that can estimate the user's emotional state through the user's facial expression data, and in this process, the usefulness of the deep learning technique is checked. Meanwhile, the user's emotional state is classified into six dimensions. Next, we develop an emotion-based category indexing recommendation method that assigns an index based on the identified user's emotional state category and makes a recommendation within the same index. Through this, the user's emotional state for the recommended product that the user encounters is identified, and the precision of the recommendation is improved by performing collaborative filtering within a group of products expressing similar emotional states.

The contribution of this study is to propose procedures and methods that can reflect the concept of emotion AI in the recommendation process. To actually implement this, it is necessary to collect large-scale real facial expression data and the user's recommendation scores linked to it, but since this is difficult to implement in a laboratory environment, this study is thought to be meaningful in proposing the procedure and method.

The structure of this study is as follows. First, Section 1 and 2 presents the motivation and the background for this study, Section 3 and 4 provides the research procedures and the results of the study, and Section 5 presents the conclusions and limitations of the study.

II. Research Backgrounds

Emotion AI, also called affective computing, develops and studies a system that can recognize, interpret, process, and simulate human emotions. It is a technology that makes it possible to understand and utilize human emotions by utilizing facial expressions, body movements, and human body information (changes in blood pressure, etc.). In this research, we develop a method to apply emotion AI to a collaborative filtering-based recommendation system by using emotion AI as contextual information in the form of predicting the user's emotional state shown in the user's facial expression for the recommended product. Human facial expressions can be said to be an important tool for expressing human emotions externally, and studies have been conducted to understand human emotions through facial expressions. In particular, the study by Ekman & Friesen [1] is known as the first systematic study to recognize human emotions through human facial expressions. In that study, the research team measured the characteristics of facial expression changes as distances or angles between coordinates in 25 types of faces (refer to a. in Fig. 1), and then calculated 41 types of AUs (action unit) (refer to b. in Fig. 1), and organized them into six emotion classification systems (refer to c. in Fig. 1). The contents claimed that various emotions that appear in human expressions can be organized into six categories: surprise, fear, disgust, anger, happiness, and sadness. There is a limit to expressing all human emotions with these 6 emotions, but other emotions can be expressed by combining the 6 basic emotion.

Fig. 1. Six Emotion Classification Systems [Excerpts from 1 and 2]

On the other hand, Russell [3] proposed a VA (valence-arousal) model, which expresses the relationship between various emotions in addition to the six emotions in a two-dimensional space by expanding the contents of the above study. Based on the two intersecting axes, emotions are expressed according to the positive-negative valence on the left and right sides, and the arousal on the upper and lower sides.

Meanwhile, in this research, deep learning techniques is used in implementing emotion AI. Specifically, deep learning techniques suitable for classification of image data are used to collect facial expression data of users viewing recommended screens and classify them into several emotion categories.

The reason why deep learning has been applied to recommendation systems in previous studies is that one of the existing recommendation algorithms, such as matrix factorization, is based on a linear model, and deep learning was often introduced to make nonlinear modeling possible. In order to consider text data such as product reviews and Twitter, or unstructured data such as images, videos, and voices, which are mass-produced in the recommendation systems, RNNs (recurrent neural networks) or CNNs (convolutional neural networks) were often used.

Sequential data is data with time series, text data is data such as product reviews and Twitter data, image data is data such as photos and pictures, and voice data is human voice, sound, music, etc. Image data refer to data such as multimedia and video, and network data refer to social network data and quotation network data. This is a summary of what kind of data deep learning was used to process in the recommendation system according to each of the above-mentioned deep learning characteristics. As described above, DNNs (deep neural networks) are used to process various data [4-10], and autoencoders are used for feature conversion of various data types, noise removal, and dimensionality reduction [11-13]. In the case of CNN, it has been used in previous studies to process text, image, and video data by utilizing the characteristics that have strengths in image, video, and audio data processing [14-21]. RNN is mainly used for processing sequential data [22-25], while RBM (restricted Boltzmann machine) and GAN (generative adversarial network) are not used much in research on recommendation systems because those techniques do not have great strengths in data processing used in recommendation systems [26]. Comprehensive of previous studies, DNN is widely used in prior research on recommendation systems. Autoencoders are mainly used for text and network data processing, CNNs for image and text data processing, and RNNs for sequential data processing. It is thought to be a good use of the characteristics of each deep learning technique.

III. Research Process

This study has two research steps as follows. The first step is to develop a deep learning-based emotion prediction module to implement emotion AI in the field of recommendation systems. Specifically, we classify the user's emotional state shown in the user's facial expression into several categories based on previous studies, and develop an algorithm that can predict the user's emotional state based on this. To this end, the user's emotional state shown in the user's facial expression is classified using the six emotional system models and the Valence-Arousal model proposed in previous studies. Afterwards, a predictive model is developed that can predict where the new user's facial expression belongs among the previously classified emotional state categories. Recently, since most online buyers make transactions in a mobile environment equipped with a camera, such as a smart phone, it has become easy to acquire information about the user's facial expression. Recently, there is a lot of human facial expression data published on the web, and after analyzing it to classify the user's emotional state, we develop an emotional state prediction model using the classification result.

The deep learning technique to be used for emotional state classification is a convolutional neural network, which is known to show good performance in image processing. In the case of convolutional neural network, the user's facial expression data is converted into a 2-dimensional matrix, and in the Convolutional Layer, filters of size (2, 3, 4, 5) are convolutionally multiplied by valid padding and stride methods. And the activation function used was Relu, which is widely used in Convolutional Neural Network. In the pooling layer, max pooling of size 2 is applied. Thereafter, the various output values of the pooling layer are flattened and connected, sent to a fully-connected layer, passed through an activation function, and connected again to a fully-connected layer having one unit, and the sigmoid activation function is applied. In this process, a dropout probability is applied to mitigate overfitting.

The second step is to predict the emotional state based on the facial expressions of new users using the developed emotional state prediction model, and based on this, to generate recommendation results from the user product matrix of users in similar emotional states. We propose category-indexing collaborative filtering. Category index collaborative filtering is an algorithm that separately creates a user-product matrix with a specific category index among all user-product matrices and creates a recommendation list using this. Fig. 2 summarizes the research progress above.

Fig. 2. Conceptual diagram of recommendation process in this study

As suggested in Fig. 2, the user's facial expression is first collected through the camera of the computer or mobile device that the user is using, and then the user's emotional state is predicted. At this time, a prediction model for predicting the emotional state is needed, and the emotion prediction module was developed. In this process, deep learning techniques such as convolutional neural networks is used. Next, each case is given a category index according to the predicted emotional state, and collaborative filtering is performed using the preference ratings of users in similar emotional states using the user-product matrix subdivided according to this index.

First of all, in order to implement the emotion prediction module, which is the first research step, consideration of related preceding studies and opinions of experts in related fields are sufficiently collected. In addition, most of the preceding studies on user emotion prediction through facial expressions judged emotion prediction as a matter of estimation and used artificial neural networks or support vector regression. Therefore, it is thought that the latest classification techniques such as deep learning techniques can be used. In this research task, we assume emotion prediction as a classification problem, define emotional state categories suitable for the recommendation system, and develop an optimized classification algorithm for emotion classification prediction according to facial expressions by applying a convolutional neural network.

The second step is a category index collaborative filtering algorithm that can generate recommendation results from the user-product matrix of other users in similar emotional states. In particular, it is necessary to classify emotional states using facial expressions into several categories and perform collaborative filtering by reconstructing the user-product matrix accordingly.

IV. Results of research

In this study, FER2013 was used as the data used to learn and verify facial expression recognition [27]. The data consists of black and white photos of 48*48 pixels, and in each image, the face is located in the center to a certain extent, so it can be used for deep learning model learning for facial expression recognition without a separate crop. The data consists of a total of 35,887 images, of which 32,298 data were used as a training set and 3,589 data were used as a test set. Each image consists of a total of 7 labels (Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral), and the number of images corresponding to each label is shown in Table 1.

Table 1. Example of image by label and data distribution

However, in the obtained data, the difference in distribution ratio between each label is large and the data for specific emotions, such as Disgust, are insufficient, so it may be difficult to expect generalized classification results. Since various problems can occur in imbalanced data, various methodologies have been used to alleviate them [28, 29]. In general, since the number of emotions such as Disgust in emotion image data is often insufficient, correction is made through augmentation.

Image Augmentation is a technique that augments the number of data by applying various transformations to the original in order to increase the amount of data. In other words, the appropriate number of augmentations was selected so that 20,000 images (a total of 140,000) for each label could be used as a training dataset, and the results are shown in Table 2.

Table 2. Results of data augmentation

In this analysis, the image augmentation method stochastically augmented data by using three augmentation methods, horizontal flip, rotation, and resized crop, which are commonly used for image data augmentation.

As data for analysis was collected to achieve the goal of recognizing facial expressions in facial images, three suitable augmentation methods were selected so that the domain of data called facial data could be maintained without being damaged. Since vertical inversion or image distortion can damage the data domain, it was determined that it would not help improve generalization performance in the process of learning a deep learning model later.

For the prepared training dataset, a model was built to classify the labels of the corresponding images through Convolution Neural Networks (CNN). CNN model optimization for the data was solved by random search according to the hyper parameter space, as shown in [30] and [31]. The model consists of five 2D convolution layers and blocks in the form of batch normalization and Max pooling layer (Average pooling layer) followed by ratio of 0.4 dropout layer, with black and white 48*48 pixels of photos as input. After passing through the convolution layers, it passes through a fully connected layer of size 384 and a 7-way softmax classifier to output classification results. The learning process was learned using the Adam optimizer, the activation function was Relu, the loss was categorical crossentropy, the learning rate was 0.001, and the learning was conducted with a batch size of 256. Fig. 3 is the result showing the structure of the overall model and parameters for each learning stage.

Fig. 3. Result by learning stage

Next, Table 3 summarizes the classification accuracy in training and test data for each emotion. Disgust, Happy, and Surprise show very high classification accuracy for both training and test datasets, but there is a slight difference in classification accuracy between training and test data in Angry, Fear, Sad, and Neutral.

Table 3. Results of Classification Accuracy

The average classification accuracy in the training data for all emotions was 90.7%, and the average classification accuracy in the test data was 67.7%. Table 4 and Table 5 show the results of classification by emotion for each of the training and test datasets.

Table 4. Results of training data

Table 5. Results of test data

As for the accuracy by emotion, Disgust, Happy, and Surprise were high, while Angry, Fear, and Sad showed relatively low accuracy. Neutral shows average accuracy, which means that the model is weak in feature extraction of facial expression images with Angry, Fear, and Sad emotions, but shows good performance in recognizing Disgust, Happy, and Surprise.

Meanwhile, using the above emotion prediction model, recommendation results can be generated from the user-product matrix of other users in similar emotional states. You can utilize the category-indexing collaborative filtering algorithm. The category-indexing collaborative filtering is an algorithm that separately creates a user-product matrix with a specific category index out of the entire user-product matrix and uses this to create a recommendation list. In this process, the preference scores for products of new users and other users must be used, so the collection of that data is also necessary. In order for this to be possible, the user's emotional state must be used as situational information and reflected in collaborative filtering, and a collaborative filtering algorithm is needed that can consider a new element called emotional state in the two-dimensional space of the existing user-product matrix.

The category-indexing collaborative filtering algorithm first classifies users into 7 emotional categories according to the predicted value of the user's emotional state and uses this as an index to perform collaborative filtering within each category. Internally, each product on the user-product matrix collaborative filtering is carried out by adding the dimension of the user's emotional state along with the star preference score. By using the model developed in this study and the emotion-based category-indexing collaborative filtering, users will be able to receive product recommendations that reflect their real-time emotional state. This could be pursued in future research.

V. Conclusions

With the activation of social network services and smart devices, the amount of information on the web is increasing exponentially, and accordingly, the academic and social demand to efficiently utilize big data on the web is also rapidly increasing. This study aims to supplement the limitations of the recommendation algorithm, which is the base technology of a recommendation system that can filter excessive information on the web and provide it as personalized information. Therefore, the results of this study will improve the recommendation performance of the existing recommendation system and will be useful in practice.

In particular, this study is expected to expand the practical use of the existing recommendation system by proposing a research model to improve the performance of collaborative filtering, which is most commonly used in recommendation system research.

On the other hand, this study also has many limitations. First, experiments and verification of category-indexing collaborative filtering based on emotion prediction results were not conducted. The data to be used here must be built in a laboratory environment, but there were difficulties in collecting data in a laboratory environment, and there is also a reliability problem with the simulated data. In this study, it is intended to propose a framework of recommendation process that can identify emotions through user's facial expression data and reflect them in the recommendation process. Therefore, the fact that the recommendation process proposed in this study was not fully implemented into an actual system is a major limitation and is believed to be something that can be supplemented in future research.

In addition, facial expression data also has a limitation in that it uses open data due to difficulties in securing actual data due to difficulties, which will be supplemented by using actual data in the future.

Lastly, we did not check the overfitting of the experimental results or the distribution of type 1 and type 2 errors. This is because the purpose of this study is to explore ways to utilize human emotions in the recommendation process, not to develop an optimized emotion recognition model, and this part can be improved in the future through emotion recognition research using facial expression data.