# Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition

###### Abstract

In this paper, we present a novel approach to automatic 3D Facial Expression Recognition (FER) based on deep representation of facial 3D geometric and 2D photometric attributes. A 3D face is firstly represented by its geometric and photometric attributes, including the geometry map, normal maps, normalized curvature map and texture map. These maps are then fed into a pre-trained deep convolutional neural network to generate the deep representation. Then the facial expression prediction is simply achieved by training linear SVMs over the deep representation for different maps and fusing these SVM scores. The visualizations show that the deep representation provides a complete and highly discriminative coding scheme for 3D faces. Comprehensive experiments on the BU-3DFE database demonstrate that the proposed deep representation can outperform the widely used hand-crafted descriptors (i.e., LBP, SIFT, HOG, Gabor) and the state-of-art approaches under the same experimental protocols.

## 1 Introduction

Facial expression is one of the most natural and preeminent way for human beings to express and communicate their emotions, opinions and intentions. Its automatic analysis and recognition has a wide range of applications, such as smart Human Computer Interface (HCI), psychology studies, entertainment etc. [37], [21]. Thus, machine-based facial expression and emotion analysis, tracking, and recognition has been extensively investigated over the last two decades [21], [25], [6].

Facial Expression Recognition(FER) methods generally can be classified from three perspectives: i.e., the data modality, the expression granularity, and the temporal dynamics. From the first perspective, they are classified into 1) 2D FER (which uses 2D gray or color face images), 2) 3D FER (which uses 3D range images, point clouds, or meshes of faces). and 3) 2D + 3D multi-modal FER (which uses both 2D and 3D facial data). From the second perspective, they are divided into 1) recognition of six basic facial human emotions, namely anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA) and surprise (SU). 2) detection and recognition of facial Action Unit (AU, e.g., brow raiser, lip tightener, and mouth stretch). From the third perspective, they are categorized into static (still images) and dynamic (image sequences) FER. In this paper, we focus on the problem of recognizing the six basic facial expressions using 2D and 3D static facial attribute maps.

In the past decades, the majority of methods dedicated to develop FER system based on 2D facial data [21]. Despite significant advances have achieved in 2D FER, it still fails to solve the two major challenges: illumination changes and pose variations [21]. Recently, with the rapid development of 3D imaging and scanning technologies, it is more and more popular to capture face scans for 3D face recognition and 3D facial expression analysis. 3D facial data can provide complete and accurate facial geometry and topology structures, which are naturally robust to poses and lighting variations. Thus, FER using 3D data (3D scans and 3D videos) has attracted wide attentions in recent years [25], [6], [1].

### 1.1 Related works

3D FER methods can be roughly divided into two categories: model-based and feature-based approaches [6], [21]. For model-based approaches, dense rigid registration and non-rigid fitting techniques are utilized to get one-to-one point correspondence among face scans, a generic expression deformable model is then generated to fit unknown faces, and their fitting parameters are finally used for expression prediction. For example, a set of training faces of several persons depicting different facial expressions were used to establish an elastically deformable model in [19]. Then, asymmetric bilinear models of unknown faces were fitted to the eigenvectors of the deformable model and their expression and identity control parameters were estimated. These parameters were used to build Maximum Likelihood classifier for final 3D FER. Gong et al. [8] suggested to learn a model to decompose the shape of an expressive face into a neutral-style basic facial shape component (BFSC) and an expression shape component (ESC). The ESC is then used to design expression features. In [38], Zhao et al. proposed to build a Statistical Facial feAture Model (SFAM) for automatic landmarking, and extract shape and texture features around landmarks for 3D FER. The main drawback of model-based approaches lies in that they require to establish dense correspondence among face scans, which is still a very challenging issue. Meanwhile, time consuming procedures like dense 3D face registration and model fitting are indispensable.

Feature-based approaches generally extract local expression features around facial landmarks based on surface geometric attributes or differential quantities. For example, 3D landmark distances [28], [29], [31], [30], 3D curves [16], geometry and normal maps [20], conformal images [36], surface normal [15] and curvatures [15], [33] are some popular 3D shape features. Feature-based approaches generally perform better than model-based ones. However, their performances are largely depend on the accuracy of 3D facial landmarking, which is a challenging task [6].

### 1.2 Motivation and contributions

Facial feature or face representation is the most important part for current state-of-the-art FER systems. In fact, the visual descriptors like SIFT, HOG and LBP, that had huge impacts in the computer vision community, have been successfully exploited for both 2D and 3D FER [2], [12], [13]. Recently, these hand-crafted descriptors have been substantially outperformed by automatically-learned deep features in extensive applications ranging from low-level restoration tasks [5], [3] to high-level object recognition tasks [4], [11].

Recently, Razavian et al. [23] demonstrated a simple yet effective recognition system using the CNN features extracted from off-the-shelf pre-trained OverFeat model [26] and the linear SVM for recognition. Surprisingly, this straight-forward recognition system consistently outperforms the finely tuned state-of-the-art systems on various databases for visual recognition tasks, such as image classification, scene recognition, fine-grained recognition, attribute detection and image retrieval.

Inspired by these results, the goal of this paper is to find further evidence of the success of CNN features for the task of 3D facial expression recognition. Our aim is to design the deep representation of 3D face scans for facial expression recognition. We represent a 3D face scan by the deep features over its geometric and photometric attributes, and futher evaluate and verify the advantages of deep facial representation on the benchmark 3D FER database. In dosing so, we are interested in the following questions: (i) Which pre-trained CNN net should be used for our 3D FER task? (ii) Which layer of this net should be best for the deep representations of facial information? (iii) What kinds of good properties does the deep representation have? (iv) How about the performance of the deep representation compared to the state-of-art hand-crafted representations?

To answer these questions, in this paper, we thus employed several pre-trained CNN nets for the deep representations of multiple facial geometric and photometric attributes (Section 3 and Section 4.1). We also visualized the fascinating deep representations to seek their secrets (Section 4.2 and Section 4.3), evaluated their performance using features in different net layers (Section 5.3), and compared them to the state-of-the-art descriptors and systems (Section 5.4). Note that the deep feature learning approach has been applied to 2D FER [22], [24], but we focus on 3D FER in this paper.

The major contributions of this paper are summarized as follows: (i) We are the first to introduce the deep representation of 3D geometric (i.e. geometry, normal, and curvatures) and 2D photometric (i.e. texture) facial attributes to the automatic 3D FER. (ii) By visualizing the deep representation of facial images, we find that the introduced deep representation can provide us a complete and discriminative representation for facial attributes. (iii) We conduct comprehensive experimental evaluations on the benchmark 3D FER database and report consistent better results compared to the shallow representations (i.e. LBP, SIFT, HOG, Gobor) as well as the state-of-the-art 3D FER systems. Our results suggest that the proposed deep representation of 3D faces achieves state-of-the-art results in 3D FER task.

## 2 Overview of the proposed approach

Figure 1 illustrates an overview of the proposed approach. Given a raw textured 3D face scan, we first run the preprocessing pipeline in [18], including nose tip detection, face cropping, and pose normalization. The preprocessed textured 3D scan is then projected to 2D plane using mesh interpolation technique to generate 2D texture map and 2.5D geometry map . The coordinates information of each geometry map are then used to estimate the surface normals and curvatures of this person, resulting in three normal component maps , , and , and one normalized curvature (i.e. shape index) map . Finally, the geometric and photometric attributes of a textured 3D face scan can be described by six types of 2D facial maps, i.e. , as shown in Fig. 1 a. Each of these maps is then fed into a pre-trained deep convolutional neural network (CNN) (Fig. 1 b) for producing their corresponding deep representations: (Fig. 1 c), where is the non-linear mapping by CNN. The linear SVM classifiers are trained and tested respectively for each type of those deep representations, and their scores are fused with a simple sum rule for the final expression classification (Fig. 1 d-f).

## 3 Attribute Maps of a 3D face

To comprehensively describe the geometric and photometric attributes of a textured 3D face scan, six types of 2D facial attribute maps, namely the geometry map , texture map (), three normal maps (, , and ), as well as normalized curvature map () are employed. The geometry and texture maps are generated by performing the preprocessing and the interpolation-based mesh projection procedures on the raw face data. The normal and curvature maps are produced by estimating the normals and curvatures of 3D facial scans on the geometry map, which will be introduced as follows.

### 3.1 Normal maps

Given a normalized facial geometry map represented by a matrix:

(1) |

where represents the 3D coordinates of point Let its unit normal vector matrix () be

(2) |

where denotes the unit normal vector of In this paper, we utilize the local plane fitting method [9] to estimate . That is to say, for each point , its normal vector can be estimated as the normal vector of the following local fitted plane:

(3) |

where represents any point within the local neighborhood of point and In this work, a neighborhood of window is used. To simplify, each normal component in equation (2) can be represented by an matrix:

(4) |

where

### 3.2 Curvature map

Similar to the local plane fitting method used for normal estimations, we explored the local cubic fitting method [7] to estimate the principle curvatures. This method assumes that the local geometry of a surface is approximated by a cubic surface patch. For robustly solving the local fitting problem, both the 3D coordinates and the normal vectors of the neighboring points of the point to be estimated are used. That is, we are fitting the following equations:

(5) |

These equations can be solved by the least squares regression and the shape operator is computed as:

Then, the eignvalues of give the two principle curvatures and at point . The normalized curvatures (i.e. shape index value) at this point is defined by:

(6) |

Figure 2 shows the facial geometric and photometric attribute maps using six types of normalized 2D facial maps for six expressions of a subject in the BU-3DFE database.

## 4 Deep representation of a 3D face

This section first gives a short introduction to the pre-trained CNN models used to generate the deep representation of a 3D face. And then intuitively visualizes the coding patterns and the saliency maps of deep representation, from which we discovered two powerful properties of deep representation for 3D face, i.e., completeness and discriminative ability.

### 4.1 The architectures of pre-trained CNNs

Caffe-Net Two pre-trained models: caffe-ref and caffe-alex, implemented by the Caffe [10] software are used. The architectures of the models are copied from the AlexNet [11], which contain eight learned layers: five convolutional and three fully connected layers. The input of the AlexNet is a fixed-size RGB image.

VGG-Net Three pre-trained deep VGG-Net models [4] with fast architecture (vgg-net-f), medium architecture (vgg-net-m), and slow architecture (vgg-net-s) and two pre-trained very deep VGG-Net models : VGG Net-D (vgg-verydeep-16) and VGG Net-E (vgg-verydeep-19) [27] are used to generate deep representation in this paper. The deep VGG-Net architectures consist of 5 convolutional layers and 3 fully-connected layers. The differences between the fast, medium, and slow architectures involve the number and receptive filed size of convolutional filters, convolution strides, spatial padding styles, and max-pooling down-sampling factors. The fully-connected layers contain two 4,096 channel layers and one final 1,000 channel soft-max layer. The very deep VGG-Net architectures contain 13 or 16 convolutional layers and 3 fully-connected layers, resulting in totally 16 (vgg-verydeep-16) or 19 (vgg-verydeep-19) weight layers. The sizes of convolutional filters are fixed to , and the numbers of filters in different layers are range from 64 to 512. The input of VGG-Net is a fixed-size RGB image. All these 7 pre-trained models are download from the research page of [4].

### 4.2 Deep representations and their visualization

A textured 3D face is represented by a set of normalized geometric and photometric attribute maps, i.e., Given a pre-trained CNN model introduced in Section 4.1, the CNN hierarchically extracts multi-layer deep features from each attribute map indexed by in . In our approach, we use the deep features of the final pooling layer just after the final convolutional layer (the detail for layer selection will be discussed in Section 5.3), which is a matrix, where is the spatial resolution and is the number of feature channels determined by the number of filters in the final convolutional layer. Taking “vgg-net-s” as an example, the deep feature matrices are with the size of dimensions for any facial map . We stack them into dimensional feature vectors, and then perform normalization procedure. These feature vectors are called the deep representation of a textured 3D face scan, and then used to learn the linear SVM classifiers for the final expression prediction.

These deep representations have intuitive meaning. Following the above example, suppose the geometry map and normal component map of a textured 3D face are fed into the pre-trained CNN “vgg-net-s”, respectively, we resize their 3D deep representation matrices () into 2D deep representation matrices () and visualize them in Figure 3. It’s easy to observe that the deep codes of faces are generally very sparse, and with very high dimensionality. This implies that the deep CNN provides a high-dimensional feature space, and each facial attribute map can be represented by a sparse code, similar to the sparse representation [17] or the widely used 2-dimensional bar code. This means that the CNN provides a complete (perhaps over-complete) representation of facial information, and we call this property as ”completeness”. This sparse coding scheme is essentially different to the traditional hand-crafted descriptors, like LBP, SIFT, etc.

### 4.3 Saliency maps for deep representations

Deep representations also have very strong discriminative ability, which can be discovered by visualizing the saliency maps for different facial expressions. In this visualization approach, we aim to visualize the importance of each pixel for the final discrimination of facial expressions. For example, for the expression of “happiness”, we visualize the saliency map for a 3D face by the importance of each image pixels contributing to the final discrimination of “happiness”.

To compute the saliency map of a 3D face w.r.t. an expression indexed by , we construct a score function for assigning this face to expression by:

(7) |

where is a deep CNN model for extracting the deep features, is a geometric or photometric attribute map for 3D face , is the extracted deep feature vector for the input map. is the weights of learned SVM classifier for expression using the training attribute map of . Obviously, the higher value of implies higher confidence in labeling this 3D face as expression , the final score fuses these confidences from different attribute maps. We next compute the gradient of score function in Eqn. (7) w.r.t. the input pixels:

(8) |

where is a pixel on the 3D face. is the gradient of deep feature w.r.t. the attribute map at pixel , which can be computed by back-propagation. Its absolute value measures the importance of pixel in labeling as expression . We call this term computed over all 3D face pixels as saliency map.

Figure 4 visualizes some examples of saliency maps for different expressions. The saliency map is re-scaled to [0, 1]. We visualize it by fusing the face texture map with a dark blue background using the saliency map as weights. The less important pixels are shown in dark blue in these maps. We observe some interesting phenomena from these maps. First, the mouth are the most salient parts for discriminating all these expressions of interest. Second, the distributions of those salient parts for all expressions are consistent with the patterns of facial shape deformations, which may spread over the whole faces with different importance. Thus, the deep CNN provides a discriminative representation for 3D face by distinguishing facial expressions using the discriminative facial parts.

## 5 Experimental results

### 5.1 The BU-3DFE dataset

We evaluate our deep representation-based 3D FER approach on the BU-3DFE [35] dataset. It consists of 2,500 textured 3D face scans of 100 subjects (including 56 females and 44 males) with different ages and races. Each subject has six basic expressions: anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA), and surprise (SU) with 4 intensity levels plus a neutral (NE) expression, a total of 25 samples.

### 5.2 Experimental settings

For fair comparison, two common experimental protocols are used. In both protocols, 720 3D face scans of 60 subjects displaying the six basic expressions with intensity of level 3 and level 4 are utilized. For Protocol I, it is required to perform 100 times 10-fold cross-validation, and 60 subjects are randomly selected for training and testing at each time [2]. For Protocol II, it is required to run 100 times 10-fold cross-validation, but the 60 subjects are randomly selected once and then fixed at all times for training and testing [8]. That is, 60 subjects are randomly divided into 10 subsets, 9 subsets (54 subjects with 648 samples) are used for training, while the remaining 1 subset (6 subjects with 72 samples) are used for testing. The experiments are repeated 10 times and the average recognition score from the 10 splits is used as the performance of a round 10-fold cross-validation.

As introduced in Section 4.1, we evaluate two types of nets: Caffe-Net, Vgg-Net and their variants, a total of 7 nets for performance evaluation.
Four hand-crafted image descriptors: MS-LBP, dense-SIFT, HOG, and Gabor are used for comparison.
Please refer to [14], [32], [12], [14], respectively for the source codes or parameters of these descriptors.
The features are normalized and then fed into the linear SVM ^{1}^{1}1http://www.csie.ntu.edu.tw/ cjlin/liblinear/ with hinge loss and regularization. The parameter is set to 1.
Score-level fusion with simple sum rule is used for all the fusion procedures.

### 5.3 Layer selection for deep representation

To verify which layer of a pre-trained deep CNN should be best to extract the deep representation of a 3D face, we generate layer-dependent deep representations for different facial attribute maps and compare their accuracies for 3D FER. In particular, three representative pre-trained nets, namely caff-alex, vgg-net-m, and vgg-verydeep-16 are used, and their results are reported in Table I, Table II, and Table III, respectively. All these results are achieved by running 10 times of 10-fold cross-validation (i.e., 100 times training and testing sessions) by Protocol I. In practice, for computational efficiency, we choose the deep features of a convolutional layer following a max-pooling layer for all nets.

% | conv1 | conv2 | conv3 | conv4 | conv5 | full6 | full7 |
---|---|---|---|---|---|---|---|

dim | 69,984 | 43,264 | 64,896 | 64,896 | 9,216 | 4,096 | 4,096 |

77.10 | 79.81 | 79.60 | 79.61 | 76.28 | 72.25 | 66.24 | |

78.65 | 80.43 | 79.43 | 79.32 | 78.25 | 76.06 | 72.11 | |

79.40 | 81.99 | 81.82 | 81.76 | 80.76 | 78.08 | 74.76 | |

79.89 | 81.46 | 80.75 | 80.38 | 78.63 | 76.47 | 73.79 | |

77.29 | 80.21 | 80.69 | 80.00 | 77.81 | 74.87 | 70.38 | |

75.64 | 81.18 | 81.11 | 80.68 | 79.51 | 77.29 | 73.17 | |

81.21 | 83.44 | 83.58 | 83.38 | 82.93 | 81.64 | 80.49 |

% | conv1 | conv2 | conv3 | conv4 | conv5 | full6 | full7 |
---|---|---|---|---|---|---|---|

dim | 279,936 | 43,264 | 6,4896 | 86,528 | 18,432 | 4,096 | 4,096 |

76.40 | 80.31 | 79.83 | 79.68 | 79.22 | 76.78 | 72.13 | |

77.26 | 81.06 | 80.31 | 80.21 | 78.93 | 77.00 | 72.79 | |

78.13 | 81.49 | 81.47 | 81.39 | 79.71 | 77.39 | 75.01 | |

78.88 | 81.46 | 80.97 | 80.57 | 79.71 | 77.43 | 74.69 | |

75.83 | 80.36 | 79.96 | 79.10 | 76.99 | 72.56 | 66.71 | |

73.72 | 81.01 | 81.22 | 80.79 | 81.03 | 76.82 | 69.58 | |

80.13 | 83.14 | 83.07 | 83.24 | 83.50 | 82.44 | 80.46 |

% | conv1-2 | conv2-2 | conv3-3 | conv4-3 | conv5-3 | full6 | full7 |
---|---|---|---|---|---|---|---|

dim | 802,816 | 401,408 | 200,704 | 100,352 | 25,088 | 4,096 | 4,096 |

65.69 | 75.65 | 79.83 | 81.18 | 79.90 | 77.68 | 73.11 | |

64.54 | 72.22 | 79.01 | 80.78 | 78.38 | 74.19 | 69.46 | |

74.15 | 76.82 | 79.93 | 82.11 | 80.25 | 76.58 | 73.06 | |

74.03 | 77.10 | 80.51 | 81.51 | 78.99 | 75.54 | 71.97 | |

69.79 | 72.78 | 79.68 | 80.93 | 74.17 | 68.17 | 62.67 | |

63.79 | 72.64 | 79.58 | 80.99 | 76.40 | 70.83 | 64.42 | |

74.51 | 78.71 | 82.28 | 83.97 | 83.64 | 82.89 | 81.40 |

Method | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|

MS-LBP | 75.80 | 76.19 | 76.60 | 75.04 | 74.32 | 68.10 | 78.48 | 79.45 | 79.42 | 79.60 |

dense-SIFT | 78.63 | 79.16 | 79.93 | 79.72 | 78.24 | 73.47 | 81.21 | 81.22 | 81.19 | 81.48 |

HOG | 79.67 | 79.82 | 79.15 | 80.14 | 77.28 | 75.16 | 81.43 | 82.01 | 81.72 | 82.01 |

Gabor | 75.86 | 77.46 | 80.49 | 79.44 | 79.12 | 76.78 | 80.86 | 81.00 | 81.67 | 82.71 |

caffe-ref | 77.43 | 79.16 | 79.20 | 78.30 | 77.82 | 78.19 | 80.40 | 80.62 | 81.70 | 82.62 |

caffe-alex | 76.51 | 77.73 | 80.16 | 78.55 | 78.16 | 79.32 | 80.13 | 80.71 | 81.66 | 82.63 |

vgg-net-f | 77.62 | 77.68 | 79.46 | 78.77 | 76.13 | 80.15 | 80.40 | 81.13 | 81.93 | 82.70 |

vgg-net-m | 79.32 | 78.63 | 79.61 | 79.54 | 76.95 | 79.79 | 80.97 | 81.58 | 82.37 | 83.20 |

vgg-net-s | 80.68 | 79.27 | 80.07 | 80.34 | 79.04 | 79.86 | 80.73 | 81.37 | 82.24 | 83.07 |

vgg-verydeep-16 | 79.96 | 77.29 | 79.87 | 79.21 | 74.61 | 75.59 | 81.19 | 82.23 | 82.69 | 83.31 |

vgg-verydeep-19 | 78.12 | 78.12 | 78.89 | 79.58 | 74.31 | 75.62 | 81.68 | 82.22 | 82.81 | 83.48 |

Method | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|

MS-LBP | 76.47 | 76.77 | 77.87 | 76.41 | 77.70 | 71.65 | 79.46 | 81.29 | 81.58 | 81.74 |

dense-SIFT | 80.29 | 79.97 | 82.35 | 80.95 | 80.28 | 75.56 | 82.62 | 82.41 | 82.60 | 83.16 |

HOG | 81.89 | 82.09 | 80.58 | 81.81 | 77.95 | 78.11 | 82.72 | 83.62 | 83.28 | 83.74 |

Gabor | 77.95 | 78.80 | 81.97 | 81.10 | 81.65 | 80.36 | 81.89 | 82.09 | 83.25 | 84.72 |

caffe-ref | 79.34 | 80.05 | 80.70 | 78.69 | 80.38 | 78.97 | 80.79 | 81.30 | 82.54 | 83.46 |

caffe-alex | 77.53 | 78.87 | 81.50 | 78.71 | 80.83 | 81.40 | 81.05 | 81.29 | 83.02 | 83.74 |

vgg-net-f | 79.67 | 79.25 | 81.66 | 79.82 | 79.05 | 82.21 | 81.58 | 82.04 | 83.42 | 84.28 |

vgg-net-m | 80.38 | 80.37 | 81.68 | 81.23 | 79.23 | 82.14 | 82.45 | 82.58 | 83.46 | 84.22 |

vgg-net-s | 80.63 | 81.70 | 80.94 | 81.90 | 80.16 | 81.44 | 81.92 | 82.19 | 83.16 | 83.91 |

vgg-verydeep-16 | 81.72 | 78.55 | 83.06 | 81.25 | 76.95 | 78.46 | 82.60 | 82.86 | 83.48 | 83.78 |

vgg-verydeep-19 | 81.04 | 80.25 | 81.10 | 82.35 | 77.33 | 79.74 | 83.58 | 83.17 | 83.96 | 84.87 |

From Table I, Table II, and Table III, we observe that the dimensions of the deep representations extracted from the convolutional layers are much higher than the ones from the fully connected layers. For example, the dimensions of the first convolutional layer (i.e., conv1) and the last fully connected layer (i.e., full7) are 69,984 vs. 4096, 279,936 vs. 4096, 802,816 vs. 4096 for caff-alex, vgg-net-m, and vgg-verydeep-16, respectively. However, the convolutional layers (except conv1) generally perform much better than the fully connected layers for all attribute maps. More precisely, the second convolutional layers (i.e., conv2) of caff-alex and vgg-net-m, and the forth convolutional layer (i.e., conv4-3) of vgg-verydeep-16 achieve the best results for nearly all the attribute maps. Note that there are two convolutional layers between conv2 and conv5 for caff-alex and vgg-net-m. Similarly, there are also two convolutional (or sub-convolutional) layers between conv4-3 and conv5-3 for vgg-verydeep-16. These results suggest us to make a trade-off between the accuracy and the efficiency.

Notice that the last convolutional layer (i.e., conv5) has the same level of deep representation dimensions (around 24,096, 44,096, and 64,096) compare with the fully connected layers (i.e., full6 or full7). Meanwhile, the performance of the conv5 are much better than the ones of full6 and full7. Moreover, when considering the score-level fusion of all the attribute maps, the accuracies achieved by conv5 are also very close to or even better than (for vgg-net-m) the ones obtained by other convolutional layers. Considering the balance between the accuracy and the efficiency, we suggest to choose the last convolutional layer (i.e. conv5) to extract the deep representation of 3D face scans for the three representative nets caff-alex, vgg-net-m, and vgg-verydeep-16. In consideration of the strong structure similarity among these representative nets and the others, we also suggest to use the last convolutional layer (i.e. conv5) to extract the deep representation of 3D face scans for other four pre-trained nets (i.e., caff-ref, vgg-net-f, vgg-net-s, and vgg-verydeep-19) used in this paper.

### 5.4 Performance evaluation and comparison

We now compare the performance of deep features to the popular hand-crafted features in vision community for 3D FER. Table 1 and Table 2 show the average recognition accuracies of four hand-crafted descriptors and seven pre-trained nets in Protocol I and Protocol II, respectively. For deep representation, we present the average accuracies for both the separate attribute maps and their score-level fusions. Based on these results shown in Tables IV and V, we have the following four conclusions:

(1) For hand-crafted descriptors, Gabor and HOG perform better than dense-SIFT and MS-LBP. When considering the contributions of all attribute maps, Gabor achieves the highest accuracies of 82.71% and 84.7% for Protocol I and Protocol II, respectively.

(2) For pre-trained nets, vgg-net-s, vgg-net-m, and vgg-net-f are comparable to each other, which are slightly better than caffe-ref and caffe-alex, and vgg-verydeep-19 obtains the best results (83.48% and 84.87% for Protocol I and Protocol II, respectively) when considering the contributions of all attribute maps. This result is consistent with the main findings in [27], i.e., better results are benefit from deeper net structure.

(3) For different attribute maps, normal maps perform slightly better than others and their fusion can generally improve the accuracy. Moreover, the fusion of all 3D geometric attribute maps (i.e., ) generally performs better than the 2D photometric attribute map (i.e., ), and their fusion (i.e., ) achieves the best performance for all nets. These results indicate that different facial attribute maps contain large complementary information for 3D FER.

(4) The results of the pre-trained nets are generally better than the results of hand-crafted descriptors, the superiorities are more obvious under the more fair comparison setting, i.e. Protocol I. Moreover, the results of Protocol II are better than the results of Protocol I, which indicates that the setting of Protocol II contain considerable subject bias and thus are not as faithful as Protocol I.

Table VI compares the average confusion matrix of Gabor and the proposed deep representation based on vgg-verydeep-19 in Protocol I. It can be seen that the proposed deep representation outperforms Gabor for all expressions except sadness and surprise with slight differences. In particular, deep representation have more powerful discriminative ability to distinguish fear expression.

% | AN | DI | FE | HA | SA | SU |
---|---|---|---|---|---|---|

Gabor | ||||||

AN | 80.01 | 4.76 | 2.62 | 0.48 | 11.94 | 0.19 |

DI | 4.96 | 81.37 | 3.66 | 2.85 | 2.72 | 4.44 |

FE | 4.51 | 5.45 | 62.84 | 14.15 | 5.64 | 7.41 |

HA | 0 | 0.43 | 3.93 | 95.17 | 0 | 0.48 |

SA | 14.56 | 0.57 | 3.35 | 0.96 | 80.54 | 0 |

SU | 0.12 | 0.96 | 1.38 | 0.95 | 0.24 | 96.34 |

deep | ||||||

AN | 82.10 | 4.58 | 2.57 | 0.48 | 10.27 | 0 |

DI | 3.64 | 82.21 | 4.67 | 2.60 | 2.64 | 4.24 |

FE | 3.27 | 7.68 | 68.19 | 11.98 | 4.72 | 4.17 |

HA | 0 | 0 | 4.14 | 95.27 | 0 | 0.58 |

SA | 18.47 | 1.14 | 2.87 | 0.34 | 77.17 | 0 |

SU | 0 | 0.67 | 2.45 | 0.90 | 0 | 95.97 |

Table VII reports the comparison results of the proposed method and the state-of-the-art 3D FER methods in the literatures of [33], [28], [30], [8], [2], [15], [13], [12], [36], [39], [34]. The early works [33], [28], [30], [2], [15] except [8] require a large set of manual facial landmarks, marking them not automatic and not practical. Recent studies use holistic registration [13], [12], [39], [34] or several automatically detected landmarks [36], making their methods to be fully automatic. Benefit from the two powerful properties of deep representation for 3D face, i.e., completeness and discriminative ability, our method achieves an average recognition rate of 83.48% and 84.87% on the BU-3DFE dataset using Protocol I and Protocol II, which outperforms the state-of-the-art ones. Note that most of the state-of-the-art methods use SVM with non-linear RBF kernel [15], [36], [39], [34] or multiple kernel learning [13], but we only use linear SVM based on a single type of deep feature.

Method | Automatic | Average acc. (%) | |
---|---|---|---|

Protocol I | Protocol II | ||

2006 Wang et al. [33] | No | - | 61.79 |

2007 Soyel et al. [28] | No | - | 67.52 |

2008 Tang et al. [30] | No | - | 74.51 |

2009 Gong et al. [8] | Yes | - | 76.22 |

2010 Berretti [2] | No | 77.54 | - |

2011 Li et al. [15] | No | - | 82.01 |

2012 Li et al. [13] | Yes | 80.14 | - |

2013 Lemaire et al. [12] | Yes | - | 76.61 |

2013 Zeng et al. [36] | Yes | 70.93 | - |

2015 Zhen et al. [39] | Yes | 83.20 | 84.05 |

2015 Yange et al. [34] | Yes | 82.73 | 84.80 |

Ours (vgg-verydeep-19) | Yes | 83.48 | 84.87 |

## 6 Conclusion and future work

In this paper, we propose a novel method for 3D FER. Geometric and photometric attribute maps are employed to describe textured 3D face scans. Deep representation of these attribute maps are then generated using deep convolutional neural networks pre-trained on the large scale image classification task. The visualization, net selection, layer selection, and performance of the proposed deep representation are studied. We find that the deep representation of 3D face have two powerful properties: completeness and strong discriminative ability. Experimental results on BU-3DFE data show that this kind of “off-the-shelf” deep representation can outperform the well known hand-crafted descriptors (i.e., LBP, SIFT, HOG, Gobor) and the state-of-the-art methods for 3D FER.

In the future, we will study the fine-tuning technique to transfer the expression feature and label information to the pre-trained deep networks. Moreover, we will also try to build a new deep CNN model for 3D FER by end-to-end training of the feature extraction in separate attribute maps, feature dimensionality reduction and fusion in a single framework. We are also interested in designing a 3D face deep representation based on facial landmarks.

## Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant No.11401464 and No.61472313, Chinese Postdoctoral Science Foundation under Grant No. 2014M560785, and Agence Nationale de Recherche (ANR) through the Jemime project under grant No. ANR-13-CORD-004-02.

## References

- [1] 3d/4d facial expression analysis: An advanced annotated face model approach. Image and Vision Computing, 30(10):738 – 749, 2012.
- [2] S. Berretti, A. Bimbo, P. Pala, B. Amor, and M. Daoudi. A set of selected sift features for 3d facial expression recognition. In 20th International Conference on Pattern Recognition, pages 4125–4128, 2010.
- [3] H. Burger, C. Schuler, and S. Harmeling. Image denoising: Can plain neural networks compete with bm3d? In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2392–2399, June 2012.
- [4] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In British Machine Vision Conference, 2014.
- [5] C. Dong, C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In Computer Vision ¨C ECCV 2014, volume 8692 of Lecture Notes in Computer Science, pages 184–199. 2014.
- [6] T. Fang, X. Zhao, O. Ocegueda, S. Shah, and I. Kakadiaris. 3d facial expression recognition: A perspective on promises and challenges. In IEEE International Conference on Automatic Face Gesture Recognition and Workshops, pages 603–610, 2011.
- [7] J. Goldfeather and V. Interrante. A novel cubic-order algorithm for approximating principal direction vectors. ACM Transactions on Graphics (TOG), 23(1):45–63, 2004.
- [8] B. Gong, Y. Wang, J. Liu, and X. Tang. Automatic facial expression recognition on a single 3d face by exploring shape deformation. In Proceedings of the 17th ACM international conference on Multimedia, pages 569–572, 2009.
- [9] R. Hoffman and A. K. Jain. Segmentation and classification of range images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(5):608–620, 1987.
- [10] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
- [11] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105. 2012.
- [12] P. Lemaire, B. Ben Amor, M. Ardabilian, L. Chen, and M. Daoudi. Fully automatic 3d facial expression recognition using a region-based approach. In Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding, pages 53–58, 2011.
- [13] H. Li, L. Chen, D. Huang, Y. Wang, and J.-M. Morvan. 3d facial expression recognition via multiple kernel learning of multi-scale local normal patterns. In 21st International Conference on Pattern Recognition (ICPR), pages 2577–2580, 2012.
- [14] H. Li, D. Huang, L. Chen, and Y. Wang. A group of facial normal descriptors for recognizing 3d identical twins. In IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems, pages 271–277, 2012.
- [15] H. Li, J.-M. Morvan, and L. Chen. 3d facial expression recognition based on histograms of surface differential quantities. Advances Concepts for Intelligent Vision Systems, pages 483–494, 2011.
- [16] A. Maalej, B. Ben Amor, M. Daoudi, A. Srivastava, and S. Berretti. Local 3d shape analysis for facial expression recognition. In Pattern Recognition (ICPR), 2010 20th International Conference on, pages 4129 –4132, 2010.
- [17] J. Mairal, F. R. Bach, and J. Ponce. Sparse modeling for image and vision processing. CoRR, abs/1411.3230, 2014.
- [18] A. S. Mian, M. Bennamoun, and R. Owens. An efficient multimodal 2d-3d hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(11):1927–1943, 2007.
- [19] I. Mpiperis, S. Malassiotis, and M. Strintzis. Bilinear models for 3-d face and facial expression recognition. IEEE Transactions on Information Forensics and Security, 3(3):498 –511, sept. 2008.
- [20] O. Ocegueda, T. Fang, S. K. Shah, and I. A. Kakadiaris. Expressive maps for 3d facial expression recognition. In IEEE International Conference on Computer Vision Workshops(ICCV), pages 1270–1275, 2011.
- [21] M. Pantic and L. J. M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Trans. Pattern Anal. Mach. Intell., 22(12):1424–1445, 2000.
- [22] Z. M. Ping Liu, Shizhong Han and Y. Tong. Facial expression recognition via a boosted deep belief network. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
- [23] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. CoRR, abs/1403.6382, 2014.
- [24] S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza. Disentangling factors of variation for facial expression recognition. In Computer Vision ¨C ECCV 2012, volume 7577 of Lecture Notes in Computer Science, pages 808–822. 2012.
- [25] G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin. Static and dynamic 3d facial expression recognition: A comprehensive survey. Image and Vision Computing, 2012. in press.
- [26] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR, abs/1312.6229, 2013.
- [27] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
- [28] H. Soyel and H. Demirel. Facial expression recognition using 3d facial feature distances. In Image Analysis and Recognition, volume 4633 of Lecture Notes in Computer Science, pages 831–838, 2007.
- [29] H. Soyel and H. Demirel. 3d facial expression recognition with geometrically localized facial features. In 23rd International Symposium on Computer and Information Sciences (ISCIS), pages 1 –4, 2008.
- [30] H. Tang and T. Huang. 3d facial expression recognition based on automatically selected features. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops(CVPRW), pages 1–8, 2008.
- [31] H. Tang and T. Huang. 3d facial expression recognition based on properties of line segments connecting facial feature points. In 8th IEEE International Conference on Automatic Face Gesture Recognition(FG), pages 1–6, 2008.
- [32] A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms, 2008.
- [33] J. Wang, L. Yin, X. Wei, and Y. Sun. 3d facial expression recognition based on primitive surface feature distribution. In IEEE Conference on Computer Vision and Pattern(CVPR), pages 1399–1406, 2006.
- [34] X. Yang, D. Huang, Y. Wang, and L. Chen. Automatic 3d facial expression recognition using geometric scattering representation. In IEEE International Conference on Automatic Face and Gesture Recognition, 2015.
- [35] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato. A 3d facial expression database for facial behavior research. IEEE International Conference on Automatic Face and Gesture Recognition, pages 211–216, 2006.
- [36] W. Zeng, H. Li, L. Chen, J.-M. Morvan, and X. D. Gu. An automatic 3d expression recognition framework based on sparse representation of conformal images. In 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1–8, 2013.
- [37] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell., 31(1):39–58, 2009.
- [38] X. Zhao, D. Huang, E. Dellandrea, and L. Chen. Automatic 3d facial expression recognition based on a bayesian belief net and a statistical facial feature model. International Conference on Pattern Recognition (ICPR), pages 3724–3727, 2010.
- [39] Q. Zhen, D. Huang, Y. Wang, and L. Chen. Muscular movement model based automatic 3d facial expression recognition. In MultiMedia Modeling,Lecture Notes in Computer Science, volume 8935, pages 522–533, 2015.