Re-learning of Child Model for Misclassified data by using KL Divergence in AffectNet: A Database for Facial Expression ©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Re-learning of Child Model for Misclassified data by using KL Divergence in AffectNet: A Database for Facial Expression thanks: ©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Takumi Ichimura Advanced Artificial Intelligence Project Research Center,
Research Organization of Regional Oriented Studies,
and Faculty of Management and Information System,
Prefectural University of Hiroshima
1-1-71, Ujina-Higashi, Minami-ku,
Hiroshima 734-8558, Japan
   Shin Kamada Advanced Artificial Intelligence Project Research Center,
Research Organization of Regional Oriented Studies,
Prefectural University of Hiroshima
1-1-71, Ujina-Higashi, Minami-ku,
Hiroshima 734-8558, Japan

AffectNet contains more than 1,000,000 facial images which manually annotated for the presence of eight discrete facial expressions and the intensity of valence and arousal. Adaptive structural learning method of DBN (Adaptive DBN) is positioned as a top Deep learning model of classification capability for some large image benchmark databases. The Convolutional Neural Network and Adaptive DBN were trained for AffectNet and classification capability was compared. Adaptive DBN showed higher classification ratio. However, the model was not able to classify some test cases correctly because human emotions contain many ambiguous features or patterns leading wrong answer which includes the possibility of being a factor of adversarial examples, due to two or more annotators answer different subjective judgment for an image. In order to distinguish such cases, this paper investigated a re-learning model of Adaptive DBN with two or more child models, where the original trained model can be seen as a parent model and then new child models are generated for some misclassified cases. In addition, an appropriate child model was generated according to difference between two models by using KL divergence. The generated child models showed better performance to classify two emotion categories: ‘Disgust’ and ‘Anger’.

Adaptive Structural Learning, Deep Belief Network, Restricted Boltzmann Machine, Kullback-Leibler Divergence, Child model, Facial Expression, AffectNet

I Introduction

Deep learning techniques have been studied as the most advanced AI researches to various kinds of problems. An optimal architecture of network is required to reach high capability, but the development at new domain is a difficult work even for a skilled researcher. The adaptive structural learning method of Deep Belief Network (Adaptive DBN) [1] has an outstanding function of determination for the network structure of Restricted Boltzmann Machine (RBM) [2, 3] which is the self-organized process by hidden neuron generation and deletion algorithm during learning phase. The algorithm monitors the training situation of some parameters at the RBM network. Adaptive DBN is the hierarchical model of RBMs where a new RBM is also automatically generated to monitor the total error of deep learning. Adaptive DBN method shows the highest classification capability for image recognition of some benchmark data sets such as MNIST [4], CIFAR-10, and CIFAR-100 [5]. The classification accuracy for training data sets has been reached almost 100% and 99.5%, 97.4%, and 81.2% for test cases, respectively. The classification ratio reported in [1] is higher than that of CNN (Convolutional Neural Network) such as AlexNet [6], GoogLeNet [7], VGG16 [8], and ResNet [9].

Recently, some databases of facial expression and emotion attracted much attentions. Mera et al. has proposed the Emotion Generating Calculations (EGC) which is a method to calculate an agent’s emotion from the contents of utterances and to express emotions which are aroused in computer agent by using synthesized facial expression [10, 11]. EGC [12] based on the Emotion Eliciting Condition Theory [13] can decide whether an event arouses pleasure or not and quantify the degree under an event. However, many facial image data were not collected to build the network with sufficient classification capability and then the trained neural network had only poor representation power.

Such models to quantify affective facial behaviors are classified into three categories: “categorical model,” “dimensional model,” and “Facial Action Coding System (FACS) model.” These databases are captured from movies or websites. In the categorical model, emotion is chosen from affective categories such as six basic emotions by Ekman et al. [14]. The categorical model cannot explain mixture of emotions by using the simple combination of emotional words such as “happily surprised”. As the second model, the dimensional model has valence and arousal. The facial images plotted in the 2D space of valence and arousal. The valence refers to how positive or negative an event is, and the arousal reflects whether an event is exciting or calm [15]. Facial Action Coding System (FACS) model has all possible facial actions in Action Units (AUs) [16]. However, FACS model does not explain the affective state directly, although it shows the facial movements.

AffectNet [17] is a kind of the dimensional model which can deal both intensity of emotion and different emotion categories in the continuous dimensional model. AffectNet contains more than one million images with faces and extracted facial landmark points. In [17], 12 human experts manually annotated 450,000 of these images in both categorical and dimensional (valence and arousal) models.

The developed deep learning model in [17] does not become to perform good classification capability (72.0%). On the contrary, Adaptive DBN shows almost 100.0% for the training data set, but it remains to be around 80.0% for the specified category of the test data because human emotions contain many ambiguous features or patterns leading wrong answer which includes the possibility of being a factor of adversarial examples [18], due to two or more annotators answer different subjective judgment for an image. In order to represent such cases, two or more child models are investigated in addition to the parent model. We developed a new child model to train only for samples that classification accuracy is not much high and investigated the Kullback-Leibler (KL) Divergence between the original parent model and the child model by Adaptive DBN. As an experimental result, we had the situation that requires an additional child model according to KL divergence. This paper discusses the relation between two or more deep learning models by using KL divergence.

The remainder of this paper is organized as follows. In section II, basic idea of the adaptive structural learning of DBN is briefly explained. The section III explains the facial database: AffectNet. In section IV, the effectiveness of our proposed method is verified on AffectNet. In section V, we give some discussions to conclude this paper.

Ii Adaptive Structural Learning Method of Deep Belief Network

The basic idea of our proposed Adaptive DBN is described to deepen understanding the sophisticated method.

Ii-a Neuron Generation and Annihilation Algorithm of RBM

While recent deep learning model has higher classification capability, the size of its network structure or the number of its parameters that a network designer must determine are larger. For the problem, we have developed the adaptive structural learning method in RBM model, called Adaptive RBM [1]. RBM is an unsupervised graphical and energy based model on two kinds of layers; visible layer for input and hidden layer for feature vector, respectively. The neuron generation algorithm of Adaptive RBM is able to generate an optimal size of hidden neurons for given input space during its training situation.

The key idea of the neuron generation is Walking Distance (WD), which is inspired from the multi-layered neural network in the paper [19]. WD is the difference between the past variance and the current variance for learning parameters. The paper [19] described that if the network does not have enough neurons to classify them sufficiently, then the WD will tend to fluctuate large after the long training process. The situation shows that some hidden neurons may not represent an ambiguous pattern due to the lack of the number of hidden neurons. In order to represent ambiguous patterns into two neurons, a new neuron is inserted to inherit the attributes of the parent hidden neuron as shown in Fig. 1(a).

In addition to the neuron generation, the neuron annihilation algorithm was applied to Adaptive RBM after neuron generation process. We had a situation that some unnecessary or redundant neurons were generated due to the neuron generation process. Therefore, such neurons can be removed according to the situation of output activation in the model after neuron generation process. Fig. 1(b) shows that the corresponding neuron is annihilated.

(a) Neuron generation
(b) Neuron annihilation
Fig. 1: Adaptive RBM
Fig. 2: An Overview of Adaptive DBN

Ii-B Layer Generation Algorithm of DBN

DBN is a hierarchical model of stacking the several pre-trained RBMs. For building process, output (hidden neurons activation) of -th RBM can be seen as the next input of -th RBM. Generally, DBN with multiple RBMs has higher data representation power than one RBM. Such hierarchical model can represent the specified features from an abstract concept to concrete representation at each layer in the direction to output layer. However, the optimal number of RBMs depends on the target data space.

We developed Adaptive DBN which can automatically adjust an optimal network structure by the self-organization by using the idea of WD. If the values of both WD and the energy function do not become small values, then a new RBM will be generated to keep the suitable network structure for the data set, since the RBM has lacked data representation capability to figure out an image of input patterns. Therefore, the condition for layer generation is defined by using the total WD and the energy function. Fig. 2 shows the overview of layer generation in Adaptive DBN.

Iii AffectNet and Facial Emotion models

AffectNet is a facial image database containing human emotions created by [17]. As shown in Table I, 11 categories are given and divided into training cases and test cases, respectively.

Fig. 3 shows the 11 categories of facial image. These categories are labeled according to human subjectivity based on valence and arousal to each facial image collected from the Internet. In [17], There is a description of the corresponding rate which two annotators make a same classification, that is, they respond a facial image as the same emotion category. The corresponding rate of two annotators’ answer is not high, because human emotions contain many vague features and each annotator can be not always to perceive same emotion. Please refer [17] for the difference of category.

The three categories ‘None’, ‘Uncertain’ and ‘Non-Face’ are not categories related to facial emotions. This paper also excludes the three categories in the same way of [17]. Moreover, the database contains the values ​​of valence and arousal, a rectangle indicating face in the image, and 68 face landmarks in an image. This paper uses valence and arousal in as shown in Fig. 4. Fig. 4 shows the distribution of valence and arousal for eight emotion categories.

Category Training Test
Neutral 74,874 500
Happy 134,415 500
Sad 25,459 500
Surprise 14,090 500
Fear 6,378 500
Disgust 3,803 500
Anger 24,882 500
Contempt 3,750 500
None 33,088 500
Uncertain 11,645 500
Non-Face 82,414 500
Sum 414,798 5500
TABLE I: The number of images in Each Category
(a) Neutral
(b) Happy
(c) Sad
(d) Surprise
(e) Fear
(f) Disgust
(g) Anger
(h) Contempt
(i) None
(j) Uncertain
(k) Non-Face
Fig. 3: Facial Images in AffectNet
Fig. 4: Valence and Arousal

Iv Experimental Results

Iv-a Classification Results

The Adaptive DBN was trained for AffectNet. Table II shows the classification results of the model. The most classified category was ‘Happy’, while the most misclassified category was ‘Anger’. For test data, the classification accuracy for ‘Anger’ was 78.4%. The categories that classification accuracy was smaller than 90.0% were ‘Neutral,’ ‘Sad,’ ‘Surprise,’ ‘Anger,’ and ‘Contempt.’ Table III shows the F1-score on AlexNet [17] and Adaptive DBN. The macro average of Adaptive DBN was 0.874. The result of Adaptive DBN was better than that of AlexNet. F1-score means the harmonic average of precision and recall as follows.


where TP is True Positive, FP is False Positive, and FN is False Negative. P is Precision and R is Recall.

Category Training set Test set
Neutral 99.3% 87.8%
Happy 99.9% 92.4%
Sad 99.2% 84.2%
Surprise 99.4% 85.8%
Fear 99.5% 90.4%
Disgust 99.3% 92.4%
Anger 98.2% 78.4%
Contempt 98.8% 87.6%
Ave. 99.3% 87.4%
TABLE II: Classification Ratio by Adaptive DBN
Category CNN[17] Adaptive DBN
Neutral 0.63 0.85
Happy 0.88 0.93
Sad 0.63 0.87
Surprise 0.61 0.88
Fear 0.52 0.90
Disgust 0.52 0.83
Anger 0.65 0.85
Contempt 0.08 0.85
Ave. - 0.874
TABLE III: F1-score
Predicted Category
Neutral Happy Sad Surprise Fear Disgust Anger Contempt

Real Category

Neutral 439 2 7 5 8 16 4 19
Happy 7 462 2 0 4 12 1 12
Sad 12 3 421 13 11 20 5 15
Surprise 15 4 10 429 11 22 0 9
Fear 10 2 10 10 452 8 3 5
Disgust 8 2 3 5 8 462 5 7
Anger 14 4 8 10 9 47 392 16
Contempt 17 8 6 3 2 21 5 438
TABLE IV: Confusion Matrix for Classification Results

We investigated the predicted category for the misclassified cases. Table IV shows the confusion matrix for the classification results in Table II. The confusion matrix means the correlation table of the predicted category and the true category for facial images. As shown in Table IV, ‘Anger’ is misclassified as ‘Disgust’. Fig. 5 shows the wrong answer cases. As shown in Fig. 5(a) to 5(c), there are wrong cases that the model answers ‘Anger’, but the correct is ‘Disgust’ category. Fig. 5(d) to 5(f) are the contrary cases.

(a) Disgust Anger
(b) Disgust Anger
(c) Disgust Anger
(d) Anger Disgust
(e) Anger Disgust
(f) Anger Disgust
Fig. 5: Wrong answer cases

Iv-B Child model and Its KL Divergence

The difference between ‘Anger’ and ‘Disgust’ was not able to be distinguished from the experimental results in Table II. The paper [17] discusses that two or more annotators answer different subjective judgment, this is, the answer depends on the person. Fig. 4 shows that there is overlap with respect to ‘Anger’ and ‘Disgust’ in the 2D space of valence and arousal.

In order to distinguish such cases leading wrong answer, to use two or more child models is effective to represent them. As opposed to general ensemble learning, we have already built the Adaptive DBN model with high classification capability in Section IV-A. Therefore, some additional child models are added to the original parent model for re-learning. Some appropriate child models are generated according to difference between two models by using KL divergence.

In this paper, we focused only two categories, ‘Anger’ and ‘Disgust’, which is emarkable difference from reported results in [17], because the other emotions were correctly classified by the parent model as mentioned in Section IV-A. The data on ‘Anger’ and ‘Disgust’ were trained by using two child models, and KL divergence showed the difference in distribution between the models. The valence and arousal for facial images with large KL were investigated. The model in Section IV-A was defined as the parent model . Table V shows the data set for the evaluation of child models. Let and be a child model to train Set 1 and Set 2, respectively. and for was compared by using KL divergence which is defined as the following equation.


where and are distribution estimated by softmax layer for parent model and child model, respectively. is an input signal. means the KL divergence for .

Table VI shows the KL divergence with Eq. (4): and for given input. As a result, was larger than , this is, model represented the wrong answer. Fig. 6 also shows similar result, but it is histogram of KL for each sample. Vertical axis and horizontal axis in Fig. 6 are the value of KL and frequency, respectively. As shown in Fig. 6, the value of is almost , while the value of has wide range with . The results showed the misclassified samples were represented according to the value of KL divergence.

According to the consideration of the KL distribution as shown in Fig. 6 and the classification results in Table VI, we make an assumption that the samples are divided by as . The samples with KL thresholds in Set 2 are plotted in valence and arousal 2D space. As shown in Fig. 7, red / blue circle is the sample with larger / smaller than KL threshold. Table VII shows the classification ratio of the re-learning child model with each KL threshold. As a result, in case of , the child model was better performance. From such observations, the deep learning model classified the facial data correctly, even for the wrong cases including two distributed categories by mixture. Good child model requires the appropriate segmentation of data distribution by using KL divergence and then we will give its condition of trade-off between KL divergence and the classification capability of child models in near future.

dataset description cases
Set 0 All data in Anger and Disgust 1,000
Set 1 Correct cases for Anger and Disgust. trains Set 1. 854
Set 2 Wrong case for Anger and Disgust. trains Set 2. 146
TABLE V: Data set for re-learning child model
Model KL
TABLE VI: KL Divergence
Classification ratio
0.20 95.8%
0.15 97.2%
0.10 95.2%
TABLE VII: Classification ratio of re-learning child model
Fig. 6: Difference between KL(P, Q1) and KL(P, Q2)
Fig. 7: Valence and Arousal in KL(P, Q2)

V Conclusion

Adaptive structural learning method of DBN has high classification capability for large image benchmark databases. In this paper, the facial image database: AffectNet was trained by Adaptive DBN. However, the deep learning model of test dataset was not able to realize perfect classification for the specified emotional categories. This paper investigated the KL divergence of Adaptive DBN models where the trained model is a parent model and the new trained model only for some cases at misclassified categories is a child model. From experimental results in this paper, we will discover a suitable threshold on the deep learning model with two or more child models by using KL divergence in future.


This work was supported by JSPS KAKENHI Grant Number 19K12142, 19K24365, and obtained from the commissioned research by National Institute of Information and Communications Technology (NICT, 21405), JAPAN.


  • [1] S.Kamada, T.Ichimura, A.Hara, and K.J.Mackin, Adaptive Structure Learning Method of Deep Belief Network using Neuron Generation-Annihilation and Layer Generation, Neural Computing and Applications, pp.1–15 (2018)
  • [2] G.E.Hinton, S.Osindero and Y.Teh, A fast learning algorithm for deep belief nets, Neural Computation, vol.18, no.7, pp.1527–1554 (2006)
  • [3] G.E.Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science (LNCS, vol.7700), pp.599–619 (2012)
  • [4] Y.LeCun, L.Bottou, Y.Bengio, and P.Haffner, Gradient-based learning applied to document recognition, Proc. of the IEEE, vol.86, no.11, pp.2278–2324 (1998)
  • [5] A.Krizhevsky: Learning Multiple Layers of Features from Tiny Images, Master of thesis, University of Toronto (2009)
  • [6] A.Krizhevsky, I.Sutskever, G.E.Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Proc. of Advances in Neural Information Processing Systems 25 (NIPS 2012) (2012)
  • [7] C.Szegedy, W. Liu, Y.Jia, P.Sermanet, S.Reed, D.Anguelov, D.Erhan, V.Vanhoucke, A.Rabinovich, Going Deeper with Convolutions, Proc. of CVPR2015 (2015)
  • [8] K.Simonyan, A.Zisserman, Very deep convolutional networks for large-scale image recognition, Proc. of International Conference on Learning Representations (ICLR 2015) (2015)
  • [9] K.He, X.Zhang, S.Ren, J.Sun, J, Deep residual learning for image recognition, Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778 (2016)
  • [10] T.Ichimura, T.Yamashita, K.Mera, A.Sato, and N.Shirahama, Emotion orientated intelligent systems, In Internet-based Intelligent Information Processing Systems, R.J.Howlett, N.S.Ichalkaranje, L.C.Jain, G Tonfoni Eds., pp.183–226, World Scientific Publishing Company (2003)
  • [11] K.Mera, T.Ichimura, Emotion Generating Method on Human - Computer Interfaces In Computationally Intelligent Hybrid Systems: The Fusion of Soft Computing and Hard Computing, Seppo J. Ovaska Eds., pp.277–312 Wiley-IEEE Press (2005)
  • [12] K.Mera, Emotion Orientated Intelligent Interface, Doctoral Dissertation, Tokyo Metropolitan Institute of Technology, Graduate School of Engineering (2003)
  • [13] C.Elliott, The Affective Reasoner: A process model of emotions in a multi-agent system, Ph.D thesis, Northwestern University, The Institute for the Learning Sciences, Technical Report no.32 (1992)
  • [14] P.Ekman, W.V.Friesen, The repertoire of nonverbal behavior : Categories, origins, usage and coding, Semiotica, vol.1, no.1, pp.49–98 (1969)
  • [15] J.A.Russell, M.Bullock, Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults, Journal of Personality and Social Psychology, vol.48, no.5, pp.1290–1298 (1985)
  • [16] P.Ekman, W.V.Friesen, Facial action coding system: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press, Palo Alto (1978)
  • [17] A.Mollahosseini, B.Hasani, M.H.Mahoor, AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild, IEEE Transactions on Affective Computing, vol.10, no.1, pp.18–31 (2019)
  • [18] A.Nguyen, J.Yosinski, J.Clune, Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images, arXiv:1412.1897 [cs.CV] (2015)
  • [19] T.Ichimura, E.Tazaki and K.Yoshida, Extraction of fuzzy rules using neural networks with structure level adaptation -verification to the diagnosis of hepatobiliary disorders, International Journal of Biomedical Computing, Vol.40, No.2, pp.139–146 (1995)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description