Multimodal Attention-Based Fusion Model for Context-Aware Emotion Recognition

Abstract

Human Emotion Recognition is an exciting topic that has been attracting many researchers for a lengthy time. In recent years, there has been an increasing interest in exploiting contextual information on emotion recognition. Some previous explorations in psychology show that emotional perception is impacted by facial expressions, as well as contextual information from the scene, such as human activities, interactions, and body poses. Those explorations initialize a trend in computer vision in exploring the critical role of contexts, by considering them as modalities to infer predicted emotion along with facial expressions. However, the contextual information has not been fully exploited. The scene emotion created by the surrounding environment, can shape how people perceive emotion. Besides, additive fusion in multimodal training fashion is not practical, because the contributions of each modality are not equal to the final prediction. The purpose of this paper was to contribute to this growing area of research, by exploring the effectiveness of the emotional scene gist in the input image, to infer the emotional state of the primary target. The emotional scene gist includes emotion, emotional feelings, and actions or events that directly trigger emotional reactions in the input image. We also present an attention-based fusion network, to combine multimodal features based on their impacts on the target emotional state. We demonstrate the effectiveness of the method, through a significant improvement on the EMOTIC dataset.

1. Introduction

For many years, human emotion recognition has been an important topic and has attracted many researchers in the computer vision field. Researchers have been trying to model and apply this ability for different applications, such as robotics, entertainment, surveillance, e-commerce, games, human-computer interaction, and more. In the computer vision field, emotion recognition is encoding emotional information in the input image or video to predict the emotion. Many attempts have considered facial expressions as the main factors for emotional perception.

Consequently, the current performance is still limited in the wild settings because the same facial movement could have different meanings in different situations [1, 2]. Some previous studies in psychology [3- 6] show that emotional perception is affected by facial expression and contextual information such as body pose, human activity, social interaction, and background context.

Inspired by these explorations in psychology, previous studies in computer vision have examined contextual information to improve performance. [7, 8] utilized multi-modal approaches to implicitly capture the contextual information from the scene fused with facial and body pose features. [9] assume that the entire set of people in an image share the social identity (emotion, etc.). [10] utilizes the context elements relationships for emotion recognition. However, the contextual information has not been fully exploited yet. The scene emotion created by the surrounding environment can shape how people perceive emotion [1, 2]. For example, people tend to feel happy when they are in a happy place (birthday party, etc.).

Another essential component is a combination/fusion mechanism on multiple input contexts. The common choices of fusion mechanism are sum or concatenation, which are considered the contribution of each modality to the final prediction. However, not all modalities are helpful and informative on the final output. For example, the emotional scene gist is crucial when the main agents stand lonely without showing any facial expressions.

This paper proposes a multi-modal and context-aware network to capture the emotional scene gist information to infer the emotion prediction. Besides, an intention-based fusion network is introduced to combine multi-modal features based on the impact of each modality on the target prediction.

The contribution of the paper is to explore the effectiveness of the emotional scene gist in the input image to infer the emotional state of the primary target. Also, an attention-based fusion network to combine multimodal features based on their impacts on the target emotional state has been newly introduced.

2. Related Works

This section may be divided by subheadings. It should provide a concise and precise description of the experimental results, their interpretation as well as the experimental conclusions that can be drawn.

2.1 Emotion Recognition in Psychological Research

The importance of contextual information for emotion recognition is well supported by previous studies in psychology. [1-3] argue that only using facial expression is not sufficient to detect the emotion of a person, since human emotion is heavily affected by different types of the context, including scene context, human activity, or social interaction. Researchers have also considered that emotional body language, human body pose and/or body motion [11-13] is another means to express emotions beside facial expression. [14, 15] shows that the presence or absence of another person affects the perceived emotional state of people. [6] suggest a psychological framework identifying three main contextual factors such as a person, situation, and culture affects our emotion perception.

2.2 Context-aware Emotional Recognition

Previous research studies are based on deep learning networks to exploit contextual information for emotion recognition. [7] and [8] introduce the two novel networks having similar architectures. Both of them are constructed in the two-stream block fashion followed by a fusion network. [8] captures facial features while [7] utilizes body features in the first stream and the other focuses on exploiting contextual information from the whole input image. [10] considers all context elements in the image and uses a Graph Convolution Network to encode context features. All previous methods assume that all of the people in the image share the same emotional state and some social identities.

2.3 Attention-based Fusion Techniques

The attention mechanism has been studied and used in convolutional neural networks for a wide range of tasks, e.g. machine translation [16], image caption generation [17], object detection [18], scene segmentation [19], etc. These mechanisms include early fusion [20], late fusion [21], or incorporate these two into a hybrid fusion. In emotion recognition, additive combinations [22,23] are the common choices for either early or late fusion. However, in the real-world, not every modality is equally reliable for every data point due to sensor noise, occlusions, etc. The contributions of each modality are not equal because of noises, variances of data features, etc. Recent works have also examined variations on more sophisticated data-driven [24], hierarchical [25,26] proposed multiplicative combination methods motivated by the idea that the contributions of each modality are different among data points.

3. Proposed Method

3.1 Emotional Scene Gist Extraction and dataset

As a human, we can perceive the emotion while watching a scene. Emotion created in the scene can be seen as an emotional feature which is beneficial to infer emotional state of main agent in the input image. To capture this feature, we pretrained a Convolution Neural Network (CNN) 𝑀_e on StockEmotion [27] dataset. StockEmotion is a large-scale emotion dataset including about 1.2 million images collected from Adobe Stock with 690 emotional classes. These emotional classes consist of four main types: emotions (disappointed, nervous, frustrated, etc.), feelings (unfortunate, severe, tranquil, etc.), actions (quarrel, threat, yell pray, etc.), and events (Christmas, Halloween, wedding funeral, nightmare, etc.). The model 𝑀_e is constructed and trained based on a proposal from [27]. For the input image 𝐼, we obtain emotional scene feature 𝑥_e by passing 𝐼 through 𝑀_e.

3.2 Body Features Extraction

The relation between body poses and emotion perception has been studied over the last decades [12, 13][28,29]. Recent studies in [30,31] suggest that body poses/gestures are beneficial to infer emotional state. To extract body-related features 𝑥_b, the visible part of the body 𝐼 is cropped and then passed through the body feature extraction module. These features contain important cues like head and pose or body appearance. This module is a CNN pre-trained with ImageNet dataset [32], which contains the category person.

3.3 Place Feature Extraction

Place category and place attribute are highly correlated with emotion expression. For example, people usually show Anticipation, Excitement and Confidence and less frequently show Sadness or Annoyance while playing sports. These observations show that some common senses knowledge patterns between places and emotions could be potentially extracted from the data. To capture place category and attribute from the scene, we trained a CNN 𝑀_p on PLACES dataset [33]. PLACES dataset is a huge dataset consisting of 10 million scene photographs and annotated with 476 scene semantic categories and attributes. The architect and training procedure is followed by a proposal in [33]. Place feature 𝑥_p is extracted by passing the input image 𝐼 through 𝑀_p.

3.4 Multimodal Attention-based Fusion

Combining all extracted features by can be beneficial to infer emotion. The common choices of fusion mechanism are sum or concatenation, which treats the contribution of each modality to the final prediction equally. However, this way can make the model confuse to recognize which information is necessary. Given the extracted feature vector for each module, we propose an attention-based fusion module to automatically fuse multimodal input by estimating the impact of these features corresponding to the final emotion. The importance of those features is computed as followed:

\(X = ( x _ { b } \oplus x _ { p } \oplus x _ { e } )\)       (1)

\(W = \operatorname { softmax } ( F ( X ) ) = [ w _ { b } ; w _ { e } ; w _ { p } ]\)       (2)

\(x _ { a } = w _ { b } * x _ { b } + w _ { e } * x _ { e } + w _ { p } * x _ { p }\)      (3)

Where ⨁ is a concatenation operator. ℱ(.) is a simple Neural Network with three outputs shown in Figure 2. W∈R^3x1 is a normalized context weights presenting for three modules body, place, and the emotional scene gist. A fusion vector x_a is combination of three inputs based on the weight matrix W.

3.5 Multimodal Loss Function

To deal with multi-label problem with an inherent class imbalance issue, we leverage the discrete category loss function proposed from [7] to calculate the loss of each modality and the fusion network. The loss is defined as follows:

\(L ( y , \hat { y } ) = \sum _ { i = 1 } ^ { 26 } w _ { j } ( \hat { y _ { 1 } } - y _ { j } ) ^ { 2 }\)       (4)

\(w _ { i } = \frac { 1 } { \operatorname { ln } ( c + p _ { i } ) }\)       (5)

Where 𝑦_i is the ground-truth label and \(\hat y _ l\) is the prediction for the 𝑖-th category. The parameter 𝑤_i is the weight assigning to each category, where 𝑝_i is the probability of the 𝑖-th category and 𝑐 is a parameter to control the range of valid values for 𝑤_i. The total loss is computed by summing all of the losses from each modality as follows:

\(L _ { \text { total } } = \alpha L _ { b } ( y , \hat { y _ { b } } ) + \beta L _ { e } ( y , \hat { y _ { e } } ) + \gamma L _ { p } ( y , \hat { y _ { p } } ) + \delta L _ { a } ( y , \hat { y _ { a } } )\)       (6)

where 𝛼, 𝛽, 𝛾, 𝛿 are scalar coefficients controlling the importance of each loss.

4. Experimental Result

4.1 EMOTIC Dataset

EMOTIC dataset [7] is a huge dataset created for context-aware emotion recognition. All of the images were collected from MSCOCO [34] and ADE20K [35] along with images downloaded from web searches. The dataset consists of 23,571 images, with 34,320 people annotated for 26 discrete emotional classes and continuous Valence, Arousal, and Dominance dimensions. The emotion categories have a wide range of emotional states, including Peace, Happiness, Esteem, Anticipation, Engagement, Confidence, Affection, Pleasure, Excitement, Surprise, Embarrassment, Sympathy, Doubt/Confusion, Disconnection, Fatigue, Yearning, Disapproval, Annoyance, Aversion, Anger, Sensitivity, Sadness, Disquietment, Pain, Fear, and Suffering. In this paper, we focus on the emotion classification problem, so only the 26 discrete categories are used in the experiments. The dataset is divided into three subsets: train, validation, and test set. For a fair comparison with other researches, we use the standard train, validation, and test splits from the original paper. Some examples of this dataset are shown in Figure. 1.

Figure 1. Qualitative Results: Illustration of attention block on randomly selected images. In the first row, given the location of the main agent, the body attention heat map clearly masks a happy face and the contribution of body information is significantly higher than the others. In the second row, the face and body pose of the main agent is not clearly shown emotion. However, the semantic context of the scene is quite peaceful, which contains the emotional information to predict the main agent’s feelings.

Figure 2. An overview of our pipeline. We use three contextual interpretations: face and body cues, place attributions, and emotional scene gist to predict the final emotion.

4.2 Implementation

Our method is implemented by using Pytorch [36] framework. For the place module and emotional scene gist extractors, we pretrain two ResNet50 [37] networks on PLACES dataset and StockEmotion dataset, respectively. For the body module, we use ResNet50 with ImageNet weights initialized while the fusion network is trained from the scratch. The body, place, and emotional scene input features are denoted by B, P, and E, respectively. To optimize the model, the Stochastic Gradient Descent optimizer is used in this experiment with a batch size of 256. The model is trained in 60 epochs with a learning rate decreasing from 0.01 to 0.0001 gradually following the Multistep learning rate strategy. For more generalization, we apply some augmentations: horizontal flip, grid distortion, shift-scale, and rotation. The model has experimented on a desktop PC with AMD Ryzen 7 2700X is equipped with a NVIDIA GTX 2080Ti GPU processor.

4.3 Results in EMOTIC Dataset

In this section, we show the results of various experiments for evaluating our model. We also show in detail the ablation studies to examine and evaluate each module in our model. Finally, we conclude with quantitative and qualitative results of the context interpretations.

4.3.1 Ablation Experiments

To highlight the importance of the emotional scene gist, we combine three kinds of interpretations and then remove them one by one. We notice that the body information is permanently retained because it contains all information of the main agents. We also demonstrate the effectiveness of the attention-based fusion by comparing this fusion with additive fusion(concatenation). The mean Average Precision(mAP) is used for evaluating the emotion recognition performance. The results of ablation experiments have been shown in Table 1.

Table 1. Performances on EMOTIC Dataset

Figure 3. Illustration of attention fusion block

To evaluate the contribution of each modality, we run our model through the test set of the EMOTIC dataset and collect the impact weights from the attention-based fusion block. Our model’s predictions are mainly based on the emotional scene gist because most of the faces, bodies, or places in the image are not very clear. In some cases where the image only contains the main agent, the contribution of body information is significantly higher than in other contexts. This explains the appearance of the tail distribution shown in Figure 4.

Figure 4. The distribution of impact weights of body, emotional scene gist, and place features on the EMOTIC test set. The importance of the emotional scene gist and body features highly outweigh the one of the place features. Notice that the sum of weights on three modalities equals 1 for a single image.

4.3.2 Qualitative and Quantitative Results

We show the qualitative results for two samples in Figure 3. To better understand what the model learned, we use GradCAM [38] to generate a heat map that indicates the influence of each area in the image corresponding to the final prediction. In the first row, the smiling face clearly expresses the happiness of the main agent so the contribution of face and body information is significantly higher than the others. In the second row, the face and body pose of the main agent is not clearly shown emotion. However, the semantic context of the scene is quite peaceful, which contains the emotional information to predict the main agent’s feelings. We also summarize the evaluation of mean Average Precisions for all the methods conducting on the EMOTIC dataset in Table 1. Performance measures of different labels appear differently because of the variance in the size of the dataset and distribution characteristics of the data itself.

Table 2. Ablation Studies: We perform ablation experiments to understand how much of each modality and fusion mechanism benefits on EMOTIC dataset. In the experiments with additive fusion mechanism, we use concatenation operation to fuse multimodal features.

5. Conclusion

In summary, this paper presents a multimodal attention-based fusion network for context-aware emotion recognition. Our model explores the emotional scene gist combining with the body (pose/gait) and places attributions. Our multimodal attention-based fusion allows us to decide the context feature should be more focused on for making a prediction based on their impacts on the target emotional state. The extensive experiments have demonstrated significant improvement in the performance of the current human emotion recognition system. Our results show that the emotional scene gist is a potential factor to automatically recognize human emotion in the wild and motivate further research in this direction