Detecting Faces Using Region-based Fully Convolutional Networks
Face detection has achieved great success using the region-based methods. In this report, we propose a region-based face detector applying deep networks in a fully convolutional fashion, named Face R-FCN. Based on Region-based Fully Convolutional Networks (R-FCN), our face detector is more accurate and computationally efficient compared with the previous R-CNN based face detectors. In our approach, we adopt the fully convolutional Residual Network (ResNet) as the backbone network. Particularly, we exploit several new techniques including position-sensitive average pooling, multi-scale training and testing and on-line hard example mining strategy to improve the detection accuracy. Over two most popular and challenging face detection benchmarks, FDDB and WIDER FACE, Face R-FCN achieves superior performance over state-of-the-arts.
Face detection plays an important role in the modern face-relevant applications. Despite the great progress made in recent years, the technical challenging of face detection still exists out of the complex variations of real-world face images. As shown in Figure 1, the visual faces vary a lot as the result of the affecting factors including occlusion on the facial part, different scales, illumination conditions, various poses of person, rich expressions, etc. Recently, remarkable advances of objection detection have been driven by the success of region-based methods [1, 2, 3, 4]. Among recent novel algorithms, Fast/Faster R-CNN [3, 4] are representative R-CNN based methods that perform region-wise detections on the regions of interest (RoIs). However, directly applying the strategy of region-specific operation to fully convolutional networks, such as Residual Nets (ResNets) , results in inferior detection performance owing to the overwhelming classification accuracy. In contrast, R-FCN  is proposed to address the problem in the fully convolutional manner. The ConvNet of R-FCN is built with the computations shared on the entire image, which leads to the improvement of training and testing efficiency. Comparing with R-CNN based methods, R-FCN proposes much fewer region-wise layers to balance the learning of classification and detection for naturally combining fully convolutional network with region-based module.
As a specific area of generic object detection, face detection has achieved superior performance thanks to the appearance of region-based methods. Previous works primarily focus on the R-CNN based methods and achieve promising results. In this report, we develop a face detector on the top of R-FCN with elaborate design of the details, which achieves more decent performance than the R-CNN face detectors [7, 8]. According to the size of the general face, we carefully design size of anchors and RoIs. Since the contribution of facial parts may be different for detection, we introduce a position-sensitive average pooling to generate embedding features for enhancing discrimination , and eliminate the effect of non-uniformed contribution in each facial part. Furthermore, we also apply the multi-scale training and testing strategy in this work. The on-line hard example mining (OHEM) technique  is integrated into our network as well for boosting the learning on hard examples.
Our key contributions are summarized below:
(1) We develop a face detection framework that takes the special properties of face into account by integrating several innovative and effective techniques. The proposed approach is based on R-FCN and is well suited for face detection, thus we call it Face R-FCN.
(2) We introduce a novel position-sensitive average pooling to re-weight embedding responses on score maps and eliminate the effect of non-uniformed contribution in each facial part.
2 Related Work
In the past decades, face detection has been extensively studied. The pioneering work of Viola and Jones  invents a cascaded AdaBoost face detector using Haar-like features. After that, numerous of works [13, 14, 15] have focused on developing more advanced features and more powerful classifiers. Besides the boosted cascade methods, several studies apply deformable part models (DPM) for face detection [16, 17, 18]. The DPM methods detect faces by modeling the relationship of deformable facial parts.
Recent progress in face detection mainly benefits from the powerful deep learning approaches. The CNN-based detectors have achieved the highest performance. [19, 20, 21] construct cascaded CNNs to learn face detectors with a coarse-to-fine strategy. MTCNN  develops a multi-task training framework to jointly learn the face detection and alignment. UnitBox  propose the intersection-over-union (IoU) loss function, to directly minimize the IoUs of the predictions and the ground-truths. Recently, several methods [7, 23, 24, 25] use the Faster R-CNN framework to improve the face detection performance.  explores the contextual information for face detection and proposes a framework achieving high performance, especially improving the accuracy of tiny faces. Most recently, [27, 28] propose to use single stage framework for face detection, with carefully designed strategies and achieve the state-of-the-art performance.
Similar to face detection, general object detection is advancing rapidly thanks to the deep learning approaches. Typical work including R-CNN , Fast R-CNN , Faster R-CNN  and their extensions [6, 29, 30]. Among these studies, R-FCN makes the detection in a nearly fully convolutional manner, which greatly enhances the efficiency of training and testing. The methods of hard example mining further help deep learning based object detection to improve the performance. In , the authors proposed an on-line hard example mining (OHEM) algorithm to improve the object detection performance. [23, 24, 21] also use hard example mining algorithms to boost the performance of face detection.
3 Proposed Approach
We improve the R-FCN framework for targeting face detection in three aspects. First, we introduce additional smaller anchors and modify the position sensitive RoI pooling to a smaller size for suiting the detection of the tiny faces. Second, we propose to use position-sensitive average pooling instead of normal average pooling for the last feature voting in R-FCN, which leads to an improved embedding. Third, multi-scale training strategy and on-line Hard Example Mining (OHEM) strategy  are adopted for training. In the testing phase, we also ensemble the multi-scale detection results to improve the performance. The details of the proposed approach are described as follows.
3.1 R-FCN Based Architecture
R-FCN  is a region-based fully convolutional network initially proposed for object detection. Unlike other region-based detectors such as Faster RCNN , R-FCN constructs a deeper fully convolutional network without increasing the speed overhead by shared computation on the entire image. R-FCN builds upon 101-layer ResNet , consists of a region proposal network (RPN) and a R-FCN module in contrast to R-CNN module that presented in Faster R-CNN .
ResNet architecture in R-FCN plays the role of feature extractor. It is common knowledge that ResNet construct a very deep network which is able to extract highly representative image features. These features hold much larger receptive field where tiny face detection can be benefited from the context information. From the feature maps that output by the fundamental ResNet, RPN generates a batch of the region of interests (RoIs) according to the anchors. These RoIs further are fed into two sibling position sensitive RoI pooling layer in R-FCN module to produce class score maps and bounding box prediction maps. In the end of R-FCN, two global average pooling are applied on both class score maps and bounding box prediction maps respectively for aggregating the class scores and bounding box predictions.
There are two major advantages that we adopt R-FCN over R-CNN. Firstly, the position sensitive RoI pooling ingeniously encodes position information into each RoI by pooling group of feature maps to a certain location of the output score maps; Secondly, without unnaturally injecting fully connected layers into ResNet architecture, the feature maps of R-FCN are trained more expressive and easier for the network to learn the class score and bounding box of faces.
Based on R-FCN, We make several effective modifications for improving detection performance. For better describing tiny faces, we introduce more anchors with smaller scales (say, from 1 to 64). These smaller anchors are very helpful for sufficiently capturing the extremely tiny faces. Besides, we set smaller pooling size for position sensitive RoI pooling to reduce redundant information, and refine the following voting scheme (average pooling) to be position sensitive average pooling, which will be described in the following section. Finally, we apply atrous convolution in the last stage of ResNet to keep the scale of feature maps without losing the contextual information in larger receptive field.
3.2 Position-Sensitive Average Pooling
In the original R-FCN work, global average pooling is adopted to aggregate the features after position-sensitive RoI pooling into a single dimension. This operation leads to the uniform contribution of each position of the face. However, the contribution of each part of the face may be non-uniformed for detection. For example, in terms of face recognition, eyes usually are paid more attentions than mouth which has been verified by experiments in . Intuitively, We believe such assumption that distinct regions on the face have different importance should also hold in face detection. Hence, we propose to perform weighted average for each area of the output of position sensitive RoI pooling in order to re-weight the region, which is called position-sensitive average pooling.
Formally, let denote the output feature maps of a position-sensitive RoI pooling layer, and denote the feature map, where denotes the size of the pooled feature map. Position-sensitive average pooling calculates the weighted average value of the feature responses to get the pooling feature from , where is denoted as:
where denotes the weight for the -th position. Note that position-sensitive average pooling can be thought as performing feature embedding on every location of responses followed by average pooling. Hence, it is very convenient to implement position-sensitive average pooling on most of the popular deep neural network frameworks.
3.3 Multi-Scale Training and Testing
Inspired by , we perform multi-scale training and testing strategy to improve performance. In the training phase, we resize the shortest side of the input to 1024 or 1200 pixels. This training strategy keeps our model being robust on detecting the target at the different scale, especially on tiny faces. On-line Hard Example Mining (OHEM)  is a simple yet effective technique for bootstrapping. During training, we also apply OHEM on negative samples and set the positive and negative samples ratio to 1:3 in each mini-batch. In the testing phase, we build an image pyramid for each test image. Each scale in the pyramid is independently tested. The results from various scales are eventually merged together as the final result of the image.
We perform evaluation on two public-domain face detection benchmarks: the WIDER FACE dataset  and the FDDB dataset . The WIDER FACE dataset has a total collection of 393,703 labeled face in 32,203 images, of which 40% are used for training, 10% for validation and 50% for testing. Specifically, the validation set and the test set are divided into three subsets (Easy, Medium, and Hard) for evaluation based on different level of difficulties, as defined in . The FDDB dataset contains 5,171 labeled faces in 2,845 images. Example images of WIDER FACE and FDDB are shown in Figure 5.
4.1 Implementation Details
Our training hyper-parameters are similar to Face R-CNN . Different from Face R-CNN, we initialize our network with the pre-trained weights of 101-layer ResNet trained on ImageNet. Specifically, we freeze the general kernels (weights of few layers at the beginning) of the pre-trained model throughout the entire training process in order to keep the essential feature extractor trained on ImageNet.
In terms of the RPN stage, Face R-FCN enumerates multiple configurations of the anchor in order to accurately search for faces. We combine a range of multiple scales and aspect ratios together to construct multi-scale anchors. These anchors then map to the original image to calculate the IoU scores with the ground truth for further picking up with following rules: First, the anchors with highest IoU score are strictly kept as positive; Second, the anchors with IoU score above 0.7 are assigned as positive; Third, If the anchors have IoU score that is lower than 0.3, they are marked as negative. The R-FCN is then trained on the processed anchors (proposals) where the positive samples and negative samples are defined as IoU greater than 0.5 and between 0.1 and 0.5 respectively. The RPN and R-FCN are both learned jointly with the softmax loss and the smooth L1 loss.
Non-maximum suppression (NMS) is adopted for regularizing the anchors with certain IoU scores. The proposals are processed by OHEM to train with hard examples. We set the 256 for the size of RPN mini-batch and 128 for R-FCN respectively. Approximate joint training strategy is applied for training in the end-to-end fashion.
We utilize multi-scale training where the input image is resized with bilinear interpolation to various scales (say, 1024 or 1200). In the testing stage, multi-scale testing is performed by scale image into an image pyramid for better detecting on both tiny and general faces.
4.2 Comparison on Benchmarks
We train our model on the training set of WIDER FACE and perform evaluation on the validation set and test set following the Scenario-Int criterion . As illustrated in Figure 3, our proposed approach consistently wins the 1st place across the three subsets on both the validation set and test set of WIDER FACE and significantly outperforms the existing results [27, 28, 26, 25, 21, 14, 32, 10]. In particular, on WIDER FACE hard subset, our approach is superior to the prior best-performing one  by a clear margin, which demonstrates the robustness of our algorithm.
There are two evaluation protocols for evaluating the FDDB dataset: one is 10-fold cross-validation and the other is unrestricted training (using the data outside FDDB for training). Our experiments strictly follow the protocol for unrestricted training.
We use the training set of the WIDER FACE dataset to train our model (denoted as Model-A in Figure 4) and compare against the recently published top approaches [28, 26, 21, 33, 34, 35] on FDDB. All of these approaches use the protocol for unrestricted training defined in . The discrete ROC curves and continuous ROC curves of these approaches are plotted in Figure 4. From Figure 4, it is clearly that Face R-FCN consistently achieves the impressive performance in terms of both the discrete ROC curve and continuous ROC curve. Our discrete ROC curve is superior to the prior best-performing method [28, 8]. We also obtain the best true positive rate of the discrete ROC curve at 1000/2000 false positives (98.49%/99.07%). For the reason that we do not optimize our method to regress the elliptical ground truth in FDDB dataset, our continuous ROC curve is lower than the first place  and slightly lower than [8, 34]. Additionally, one of the factors that may affect the performance of Face R-FCN demonstrated in the last row of 5: the false positive bounding boxes in images exactly contain faces from human perspective where these faces have not been annotated as ground truth. This factor partly leads to the lower performance comparing with [8, 34, 28]. But the competitive result we achieved is still noticeable.
Furthermore, We expand the training dataset by augmenting with a privately collected dataset and use the enlarged dataset to train a more discriminative face detector (denoted as Model-B). The discrete and continuous ROC curves of Model-B are also plotted in Figure 4. As expected, the performance of Face R-FCN is further improved. Finally, we obtain the true positive rate 98.99% of the discrete ROC curve at 1000 false positives and 99.42% at 2000 false positives, which are new state-of-the-art among all the published methods on FDDB.
Face detection is a fundamental problem in vision task. In this technical report, we propose a powerful face detection approach named Face R-FCN by integrating R-FCN and several sophisticated techniques for better detecting faces and boosting overall performance. By reasoning the drawbacks of R-CNN and R-FCN, we explore the details and invent new designs to improve the popular detection framework specifically for face detection. The proposed approach is evaluated on the challenging WIDER FACE dataset and FDDB dataset. Our experimental results demonstrate the superiority of our approach over the state-of-the-arts. These innovations are inspired from past experience and we expect our innovations will be easy to generally applied to the future face detection architectures as past experience.
- R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
- R. Girshick, J. Donahue, T. Darrell, and J. Malik. Region-based convolutional networks for accurate object detection and segmentation. Transactions on Pattern analysis and machine intelligence (TPAMI), 38(1):142–158, 2016.
- R. Girshick and J. P. N. Fotheringham-Smythe and G. Gamow. Fast R-CNN. In International Conference on Computer Vision (ICCV), 2015.
- S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), 2015.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems (NIPS), 2016.
- H. Jiang and E. Learned-Miller. Face detection with the Faster R-CNN. In arXiv preprint arXiv:1606.03473, 2016.
- Hao Wang, Zhifeng Li, Xing Ji, and Yitong Wang. Face R-CNN. In arXiv preprint arXiv:1706.01061, 2017.
- A. Shrivastava, A. Gupta, and R. Girshick. Training Region-based Object Detectors with Online Hard Example Mining. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- S. Yang, P. Luo, C.-C. Loy, and X. Tang. Wider face: A face detection benchmark. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- V. Jain and E. Learned-Miller. FDDB: A benchmark for face detection in unconstrained settings. In Technical Report UMCS-2010-009, 2010.
- P. Viola and M. J. Jones. Robust real-time face detection. International Journal of Computer Vision (IJCV), 57(2):137–154, 2004.
- M. Pham, Y. Gao, V. Hoang, and T. Cham. Fast polygonal integration and its application in extending haarlike features to improve object detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
- B. Yang, J. Yan, Z. Lei, and S. Z. Li. Aggregate channel features for multi-view face detection. In International Joint Conference on Biometrics (IJCB), 2014.
- Q. Zhu, M.-C. Yeh, K.-T. Cheng, and S. Avidan. Fast human detection using a cascade of histograms of oriented gradients. In Conference on Computer Vision and Pattern Recognition (CVPR), 2006.
- R. Ranjan, V. M. Patel, and R. Chellappa. A deep pyramid deformable part model for face detection. In International Conference on Biometrics Theory, Applications and Systems (BTAS), 2015.
- X. Zhu and D. Ramanan. Face detection, pose estimation, and landmark localization in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
- M. Mathias, R. Benenson, M. Pedersoli, and L. Van Gool. Face detection without bells and whistles. In European Conference on Computer Vision (ECCV), 2014.
- H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua. A convolutional neural network cascade for face detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
- H. Qin, J. Yan, X. Li, and X. Hu. Joint training of cascaded cnn for face detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- K. Zhang, Z. Zhang, Z. Li and Y. Qiao. Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks. Signal Processing Letters, 23(10):1499–1503, 2016.
- J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang. Unitbox: An advanced object detection network. In Proceedings of the 2016 ACM on Multimedia Conference (ACM MM), 2016.
- S. Wan, Z. Chen, T. Zhang, B. Zhang, and K. Wong. Bootstrapping face detection with hard negative examples. In arXiv preprint arXiv:1608.02236, 2016.
- X. Sun, P. Wu, and S. Hoi. Face Detection using Deep Learning: An Improved Faster RCNN Approach. In arXiv preprint arXiv:1701.08289, 2016.
- C. Zhu, Y. Zheng, K. Luu, and M. Savvides. CMS-RCNN: Contextual multi-scale region-based cnn for unconstrained face detection. In arXiv preprint arXiv:1606.05413, 2014.
- P. Hu and D. Ramanan. Finding Tiny Faces. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Mahyar Najibi, Pouya Samangouei, Rama Chellappa, and Larry Davis. SSH: Single Stage Headless Face Detector. In International Conference on Computer Vision (ICCV), 2017.
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. S3FD: Single Shot Scale-invariant Face Detector. In International Conference on Computer Vision (ICCV), 2017.
- T. Lin, P. DollÂ¨Â¢r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature Pyramid Networks for Object Detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- K. He, G. Gkioxari, P. DollÂ¨Â¢r, and R. Girshick. Mask R-CNN. In International Conference on Computer Vision (ICCV), 2017.
- Aleix M Martínez. Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. Transactions on Pattern analysis and machine intelligence (TPAMI), 24(6):748–763, 2002.
- S. Yang, P. Luo, C. C. Loy, X. Tang. From Facial Parts Responses to Face Detection: A Deep Learning Approach. In International Conference on Computer Vision (ICCV), 2015.
- E. Ohn-Bar and M. Trivedi. To Boost or Not to Boost? On the Limits of Boosted Trees for Object Detection. In International Conference on Pattern Recognition (ICPR), 2016.
- Y. Li, B. Sun, T. Wu, and Y. Wang. Face Detection with End-to-End Integration of a ConvNet and a 3D Model. In European Conference on Computer Vision (ECCV), 2016.
- Open source code and models. MXNet. https://github.com/tornadomeet/mxnet-face#face-detection/, 2016.