Context-aware Cascade Attention-based RNN for Video Emotion Recognition
Emotion recognition can provide crucial information about the user in many applications when building human-computer interaction (HCI) systems. Most of current researches on visual emotion recognition are focusing on exploring facial features. However, context information including surrounding environment and human body can also provide extra clues to recognize emotion more accurately. Inspired by “sequence to sequence model” for neural machine translation, which models input and output sequences by an encoder and a decoder in recurrent neural network (RNN) architecture respectively, a novel architecture, “CACA-RNN”, is proposed in this work. The proposed network consists of two RNNs in a cascaded architecture to process both context and facial information to perform video emotion classification. Results of the model were submitted to video emotion recognition sub-challenge in Multimodal Emotion Recognition Challenge (MEC2017). CACA-RNN outperforms the MEC2017 baseline (mAP of 21.7%): it achieved mAP of 45.51% on the testing set in the video only challenge.
Understanding human emotion has attracted a lot of attention recently. And it also plays an important role in many applications such as human-computer interaction, advertising, social media communication and cognitive science. However, emotion recognition is still a challenging task. It’s very difficult to find an effective model for emotion and facial expressions, let alone combining of multimodal data of visual, vocal and even text to emotion recognition. In this work, a novel model is proposed to consider facial and context information concurrently, which leads to superior performance in emotion recognition.
One of the key problems of emotion recognition is emotion representation or emotion model. There are many researches about emotion representation, such as six discrete basic emotion classes proposed by Ekman , continuous dimensional models (e.g. valence and arousal in ), and facial action coding system (FACS) in , which describes facial movement in action units (AU). Combination of facial action units can be classified into different emotions.
Many emotion datasets and challenges have also been published using aforementioned emotion representation models such as AFF-Wild , which uses valence and arousal space to model facial expression. For the emotion classification representation, challenges such as Emotion Recognition Challenge in the Wild (EmotiW)  and Multimodal Emotion Recognition Challenge (MEC) , which is the challenge of this work attending, have been held for recent years. Furthermore, some challenges such as [5, 6] also includes vocal features for emotion recognition.
In psychological researches [7, 8], it has been discussed that other than facial expressions, contextual information such as body, pose and surrounding environment can also provide important clues for emotion perception. Evidence and experiments are also provided in [7, 8] to show that emotion perception can be influenced by context. Moreover, in some cases, context is even indispensable for emotion communication. Similar results are also proposed in computer vision literatures. Experiments in  show that when using both context and body information, performance of emotion recognition outperforms that of using only body image or only context image. A dataset “Emotions in Context Database” (EMOTIC) has also been published recently in .
Most of recent emotion recognition methods focus on exploring facial features based on deep neural network. Convolutional Neural Network (CNN) has been used to extract face features in some works [10, 11]. Some researches incorporate 3D Convolutional Networks (C3D) and Recurrent Neural Network (RNN) to model spatial and temporal clues of faces [12, 13]. Also some works [13, 12, 14] combine audio models to perform multimodal emotion recognition.
This work focuses on video emotion classification using sequence of both facial and context information. A Context-Aware Cascade Attention-based RNN is proposed here to leverage both facial and context features to perform video emotion recognition. To further evaluate the influence of context information, multiple RNN-based networks are designed to compare different methods of fusing face and context features.
Ii Related Works
Many works proposed video recognition methods based on neural networks. C3D networks were utilized to learn temporal information from sequential images for action recognition [15, 16] and video emotion recognition . Recurrent neural network, which has temporal recurrence of latent variables, was proposed in . Long-Short Term Memory (LSTM) was proposed in  to handle long-range sequence learning.
“Sequence-to-sequence” model , which models sequence encoding and generation using LSTM RNNs, is one of the popular architectures of learning from sequential data. It has achieved the state-of-the-art results in neural machine translation. The sequence-to-sequence model consists of a RNN-based encoder and a decoder to learn temporal structures and generate output sequence. Furthermore, temporal attention mechanism has also been proposed in  as a soft-alignment method in machine translation to learn relevant temporal segments from encoder and decoder.
Iii Proposed Method
In this section, the effect of context on improving emotion perception is first shown in Section. III-A. Then, inspired by Sequence-to-sequence model and attention mechanism for neural machine translation, a novel architecture for sequence classification will be proposed. Sequence-to-sequence model and attention mechanism will be briefly described in Section. III-B and III-C respectively. Then the proposed model will be introduced in detail in Section. III-D.
Iii-a Effect of Context Information on Emotion Perception
To visualize the effect of context information to emotion perception for human, an experiment was designed to show disagreement of human label of the same dataset with and without context information. The dataset consists of 55504 images with bonding box of each face. Each image is labeled by intensity of 9 emotions: “angry”, “disgust”, “happy”, “sad”, “surprise”, “fear”, “neutral”, “contempt”, “confused” and intensity is in score from 0 to 5 by each labeler. Each image is labeled by at least 5 labelers. Then the disagreement of each emotion is calculated by averaging the variance of labeled intensity of each image.
In Fig. 1, it is shown that when providing the labelers with original images of both face and context, more consensus can be observed in the obtained labels. Furthermore, emotion class “happy” and “neutral” have the least disagreement, which means happy and neutral can be recognized easily, while other emotions of higher disagreement shows more confusion in labelers.
Iii-B Sequence-to-Sequence Framework
Sequence-to-sequence, which is an encoder-decoder framework, was proposed in  to perform machine translation. The encoder reads a sequence of input vectors over input time as , into a “context vector” at encoder . The context vector here represents the stored state and information of the encoder. When the encoder is a RNN, the hidden states are represented as and , where and are nonlinear functions.
The decoder then generates one prediction at output time , , from context vector and all the previous predicted output. With a RNN decoder, its conditional probability models as
where is the hidden states of the decoder RNN at output time , , and , are nonlinear functions.
Iii-C Attention Mechanism
Attention mechanism has been proposed to improve the performance of English-to-French machine translation in  as a sequence-to-sequence model. The mechanism makes decoder not only depend the context vector from encoder but allow the decoder model to learn relevant parts over source sequence to predict target.
The soft dot global attention in  is adapted here. An alignment vector is a sequence over input time, where is derived by encoder’s hidden states at input time and current decoder’s hidden state at output time , which is
The score function implemented here is dot operation from and , where and noted that the hidden size of encoder and decoder should be equal. Then the context vector at output time is derived as weighted sum of all source and , which is computed as
Iii-D Context-Aware Cascade Attention-based RNN
Context-Aware Cascade Attention-based RNN (CACA-RNN) is proposed to leverage both face and context features to perform emotion recognition. The CACA-RNN consists of two LSTMs in a hierarchical cascade architecture with attention mechanism (Fig. 2).
Unlike encoder-decoder framework, which the decoder is a generative model, there are two encoder-like RNNs in CACA-RNN, “context RNN” and “face RNN”, which they read face feature and context feature respectively. The two RNNs are cascaded with attention mechanism in between. The attention mechanism of the CACA-RNN can locate relevant context information from the context RNN when processing the face sequence in the face RNN. After the context RNN reads whole context features, the face RNN encodes face features and its LSTM context vector is derived from attention operation of face and context RNN’s hidden states. Then the face RNN outputs the final prediction class after forwarding whole sequence.
The context RNN reads the context feature sequence from time step to as into a LSTM context vector , where its hidden states at time being denoted as . Similarily, the face RNN reads face feature sequences and its conditional probability is modeled as
Iv Comparison architectures
To investigate performance of RNN-based models fusing face and context features on emotion recognition, various architectures of combining both features or using only one of the features are introduced as the following:
Context-RNN performs emotion classification using video frames as input, depicted in Fig. 2(a). The model is similar to a model in  for action recognition, where a LSTM processes sequential features from CNN and learns to predict human action. The Context-RNN reads context features which are extracted from a CNN feature extractor, and last prediction is taken as output.
Face-RNN, on the contrary to the Context-RNN, is only using face feature sequence to predict emotion. The architecture is depicted in Fig. 2(a).
Parallel-RNN, illustrated in Fig. 2(b), consists of two RNNs processing face and context features individually. Then the hidden states of the two RNNs are fused for final prediction at latest time step.
Concatenated-RNN contains a RNN taking concatenated face and context features as input. Then its output at the last time step is taken to be the prediction of the video clip. (Fig. 2(c))
CACA-RNN A, Context-Aware Cascade Attention-based RNN, consists of two RNN in a cascade architecture with attention mechanism (Fig. 2). The left RNN reads context stream into a context vector. And the right RNN takes attention-processed context vector and face stream as input, then predicts output at the latest time.
CACA-RNN B has the same structure as CACA-RNN A (Fig. 2) except for its left RNN reads face stream and the right RNN reads context stream.
V Experiments and Evaluation
Chinese Natural Audio-Visual Emotion Database (CHEAVD) 2.0 is used by Multimodal Emotion Recognition Challenge (MEC) 2017 challenge. CHEAVD 2.0 includes 4917 training clips, 707 validation clips and 1406 testing clips from Chinese movies and TV programs. Each clip is labeled with one emotion category according to both video and audio content in 8 classes: “happy”, “sad”, “angry”, “surprise”, “disgust”, “worried”, “anxious”, and “neutral”.
To enlarge the training set, the submitted model was trained with additional private dataset. The dataset includes 4562 video clips (“happy”: 409, “sad”: 951, “angry”: 212, “surprise”: 357, “disgust”: 270, “worried”: 58, “anxious”: 88 , and “neutral”: 359) and training data clips are trimmed from the videos.
V-B Feature Extraction
For each video clip, two streams are generated in preprocessing stage: “face stream” and “context stream”. Face stream contains sequence of each detected faces from each frame, cropped and scaled to . With each detected face, corresponding context stream contains the original frame, center-cropped and scaled to .
Then, two pre-trained CNNs are used to extract features from the face and context streams as the input of CACA-RNN’s face RNN and context RNN. The face feature extractor is a classifier trained with a private dataset for image emotion classification without last layer. And the context feature extractor is a pre-trained model from Squeezenet  with ImageNet classification , which the output of the last convolution layer is used as context feature.
V-C Training Details
Videos are downsampled to fps in both training and inference phases. In training phase, each video is sampled from a random initial frame for data augmentation. In evaluation phase, the initial frame is fixed to be the first frame of video clip.
For the submitted result, Adam  was used with learning rate with mini-batch size . The weights of feature extractor CNNs are fixed while training. There is a pooling layer after the context CNN to down sample feature size to size as the input of context RNN. Also, a fully connected layer is added after the face CNN to encode face features with vector of size . Additionally, the face RNN is a two-layer LSTM with hidden states. The context RNN is also a LSTM with hidden states and attention mechanism is on face RNN.
|Model||#params||mAP (%)||ACC (%)|
V-D Quantitative results
To make comparison fair, the models are adjusted such that number of parameters are close. For each experiment, training was repeated five times with different random seeds and median performance metrics are selected among them. The feature sizes of face feature and context feature are encoded to by fully-connected layers. Face-RNN, Context-RNN and Concatenated-RNN has a two-layer LSTM with hidden nodes. In Parallel-RNN, there are two two-layer LSTMs of hidden nodes. And the first LSTM of CACA-RNN also has two layers with hidden states, the second RNN is a LSTM with hidden nodes.
Notably, the models fusing both context and face clues outperform the models using only one of the features. It shows that the context information is helpful but not enough to perform emotion recognition without cropped face.
For models fusing face and context features, the results show that Parallel-RNN outperforms Concatenated-RNN. Learning from multiple different feature sequences can be more effective for networks with multiple branches than a single branch model. The performance of CACA-RNN A and CACA-RNN B is higher than the others in evaluation in mAP (mean average precision) and accuracy. It shows that cascade attention-based RNN can improve the performance of handling two kinds of features.
Vi Results on MEC 2017
Five submissions are allowed to submit to video-based emotion recognition sub-challenge in MEC2017. Evaluation on mean average precision (mAP) and accuracy are both considered in MEC, two (of five) evaluation results of submissions are shown in Table. II: (1) “CACA-RNN”, described in previous sections, the one with highest mAP, (2) “CACA-RNN+2D-3D-CNN”, the one with highest best accuracy. Their confusion matrices are shown in Fig. 5.
CACA-RNN+2D-3D-CNN is an ensemble network of CACA-RNN and a 2D-3D-CNN network using linear weighted output from each model as the output prediction. The 2D-3D-CNN network is a spatiotemporal convolutional network, depicted in Fig. 4, learning temporal information from high level face feature maps from the same face feature extractor CNN in CACA-RNN.
|Method||Validation Set||Testing Set|
|mAP (%)||ACC (%)||mAP (%)||ACC (%)|
In this work, context information is first shown to be helpful in emotion perception. Then, different architectures using facial and/or context information for video emotion recognition was implemented and evaluation results were compared. It shows that the models using both information can achieve the highest accuracy.
Among them, a novel architecture, CACA-RNN, was proposed with a cascaded LSTM attention-based architecture to leverage both face and context information from video. CACA-RNN has the best performance in the compared models. CACA-RNN consists of two LSTMs, context RNN and face RNN, processing context and face features respectively, and attention mechanism in the face RNN enables it to learn relationship to the context RNN and fuse information from two sequences. Compared to a multi-branch RNN (Parallel-RNN) and a single RNN handling concatenated features (Concatenated-RNN), CACA-RNN outperforms on evaluation MEC2017 validation dataset. The experiments also shows that context information improves for video emotion recognition. The models with additional context features perform better than the models using only face features.
CACA-RNN may be further extended to fusing more kinds of inputs for multi-modal emotion recognition or be extended to other video-based tasks such as action recognition.
-  P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion,” Journal of personality and social psychology, vol. 17, no. 2, p. 124, 1971.
-  J. A. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, 39, 1980, pp1161-1178.
-  P. Ekman and W. V. Friesen, “Facial action coding system,” 1977.
-  S. Zafeiriou, M. Nicolao, I. Kotsia, F. Benitez-Quiroz, and G. Zhao, “Aff-wild: Valence and arousal in-the-wild challenge,” In IEEE CVPR Workshop, 2017.
-  A. Dhall, R. Goecke, S. Ghosh, J. Joshi, J. Hoey and T. Gedeon, “From individual to group-level emotion recognition: EmotiW 5.0,” ACM ICMI 2017.
-  Y. Li, J. Tao, B. Schuller, S. Shan, D. Jiang, and J. Jia, âMEC 2017: the multimodal emotion recognition challenge,â In the first Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia 2018), May, 2018.
-  U. Hess and S. Hareli, “The influence of context on emotion recognition in humans,” 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2015, pp. 1-6.
-  L. F. Barrett, B. Mesquita and M. Gendron, “Context in emotion perception,” Current Directions in Psychological Science, 2011, pp.286â290.
-  R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Emotion recognition in context,” In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
-  M. Liu, R. Wang, S. Li, S. Shan, Z. Huang and X. Chen, “Combining multiple kernel methods on Riemannian Manifold for emotion recognition in the wild,” ACM ICMI, 2014.
-  A. Yao, D. Cai, P. Hu, S. Wang, L. Sha, and Y. Chen, “HoloNet: Towards robust emotion recognition in the wild,” Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016, pp472â478.
-  S. Pini, Stefano, O. Ben-Ahmed, M. Cornia, L. Baraldi, R. Cucchiara and B. Huet, “Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild,” 10.1145/3136755.3143006, 2017.
-  Y. Fan, X. Lu, D. Li, and Y. Liu, “Video-based emotion recognition using CNN-RNN and C3D hybrid networks,” in Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, 2016, pp. 445â450.
-  P. Hu, D. Cai, S. Wang, A. Yao, and Y. Chen, “Learning supervised scoring ensemble for emotion recognition in the wild,” Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI), 2017, pp553-560.
-  A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F. F. Li, “Large-scale video classication with convolutional neural networks,” In Proc. CVPR, 2014.
-  D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” arXiv:1412.0767, 2014.
-  D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” DTIC Document, Tech. Rep., 1985.
-  S. Hochreiter and J. Schmidhuber, “Long short-term memory,” in Neural Computation, MIT Press, 1997.
-  I. Sutskever, O. Vinyals, and Q. Le, “Sequence to sequence learning with neural networks,” In Advances in Neural Information Processing Systems (NIPS), 2014.
-  D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” In ICLR, 2015.
-  C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,” In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
-  M. T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” In EMNLP, 2015.
-  J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” arXiv preprint arXiv:1411.4389, 2014.
-  J. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” In CVPR, 2015.
-  I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” In Advances in Neural Information Processing Systems (NIPS), 2014.
-  L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville, “Describing videos by exploiting temporal structure,” arXiv:1502.08029v4, 2015.
-  F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters andÂ¡ 0.5 mb model size,” arXiv preprint arXiv:1602.07360, 2016.
-  O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, Alexander C. Berg, and F. F. Li, “Imagenet large scale visual recognition challenge. International Journal of Computer Vision,” 115(3):211â252, 2015.
-  Kingma, Diederik P. and Ba, Jimmy, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2014.