FakeSpotter: A Simple Baseline for Spotting AI-Synthesized Fake Faces

FakeSpotter: A Simple Baseline for Spotting AI-Synthesized Fake Faces

Run Wang Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab
   Lei Ma Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab
   Felix Juefei-Xu Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab
   Xiaofei Xie Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab
   Jian Wang Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab
   Yang Liu
Nanyang Technological University, Kyushu University,
Alibaba Group, Xiaomi AI Lab

In recent years, we have witnessed the unprecedented success of generative adversarial networks (GANs) and its variants in image synthesis. These techniques are widely adopted in synthesizing fake faces which poses a serious challenge to existing face recognition (FR) systems and brings potential security threats to social networks and media as the fakes spread and fuel the misinformation. Unfortunately, robust detectors of these AI-synthesized fake faces are still in their infancy and are not ready to fully tackle this emerging challenge. Currently, image forensic-based and learning-based approaches are the two major categories of strategies in detecting fake faces. In this work, we propose an alternative category of approaches based on monitoring neuron behavior. The studies on neuron coverage and interactions have successfully shown that they can be served as testing criteria for deep learning systems, especially under the settings of being exposed to adversarial attacks. Here, we conjecture that monitoring neuron behavior can also serve as an asset in detecting fake faces since layer-by-layer neuron activation patterns may capture more subtle features that are important for the fake detector. Empirically, we have shown that the proposed FakeSpotter, based on neuron coverage behavior, in tandem with a simple linear classifier can greatly outperform deeply trained convolutional neural networks (CNNs) for spotting AI-synthesized fake faces. Extensive experiments carried out on three deep learning (DL) based FR systems, with two GAN variants for synthesizing fake faces, and on two public high-resolution face datasets have demonstrated the potential of the FakeSpotter serving as a simple, yet robust baseline for fake face detection in the wild.

1 Introduction

With the remarkable development of AI technologies, particularly GANs, seeing is not believing in nowadays. GANs such as ProgressiveGAN (PGGAN) [13], StyleGAN [14], and StarGAN [3] assist common users without an expert knowledge in photography editing to create synthesized photos with high quality. Specifically, the synthesized faces with GANs are becoming more and more naturally and realistically. Even our humans can be easily fooled by these synthesized fake faces. Figure 1 presents some synthesized fake faces with state-of-the-art GANs. Nobody is likely to notice that these faces in Figure 1 are all fake at first glance. These fake faces are even indistinguishable to our humans.

Figure 1: Fake faces synthesized with GAN variants with facial attributes manipulated. Faces in the first and the third row are synthesized with PGGAN [13], while faces in the second row are synthesized with StyleGAN [14]

The AI-synthesized fake faces not only bring fun to users but also raise concerns and even panics to everyone including celebrities and politicians, etc. Some tools provide interesting functionalities to users with faces synthesis techniques, for example, users can transform facial expressions or swap faces on mobile devices with just one tap (e.g., FaceApp111https://www.faceapp.com, Reflect222https://reflect.tech, and ZAO333https://apps.apple.com/cn/app/zao/id1465199127). Unfortunately, malicious abusing AI in synthesizing images also raises significant concerns like creating fake pornography [5], where a victim’s face can be naturally swapped into a naked body. A recent work [38] could swap faces with only several photos in high quality. Politicians also confused the synthesized fake faces, for example, a fake official statement may be announced with realistically facial expressions by adopting the face synthesis techniques. While these fakes are extremely difficult to be identified even by humans and easily propagate in social networks, it is urgent to call for effective techniques to detect AI-synthesized fake faces.

More recently, researchers study AI-synthesized fake faces detection in two major ways, namely image forensics-based [22, 20, 32, 15] and learning-based approaches [6, 11, 21]. The image forensics-based approaches mainly borrow ideas from traditional image forensics studies. Pixel or color level statistical features are extracted for identifying the forgery images. While the learning-based approaches consider fake detection as a binary-classification problem, new Deep Neural Network (DNN) models or loss functions are designed to differentiate real and fake samples. Unfortunately, these two approaches are vulnerable to attacks in adversarial settings. Image forensics-based approaches are prone to be affected by adding noises into images, while the learning-based approaches are principally DNNs which are easily subjected by adversarial examples. Other research utilizes photo EXIF metadata to determine whether an image is fake or real [12], however, the EXIF metadata can be easily removed or crafted by attackers with current techniques.

Existing deep FR systems (e.g., VGG-Face [24], OpenFace [2], and FaceNet [27], etc.) with well designed DNN architectures and loss functions show excellent performance in representing faces and achieve incredible progress in face recognition and verification, but it failed in spotting AI-synthesized fake faces. In DNNs, a neuron learns the representations of inputs layer-by-layer and may capture more subtle features for differentiating real and fake faces. We explore the power of neurons in representing the fine-grained features of faces in deep FR systems for hunting the differences between real and AI-synthesized fake faces. In this paper, the AI-synthesized fake faces mean the faces are synthesized with GANs unless a particular declaration.

In this paper, we propose a novel neuron coverage-based approach, named FakeSpotter, for spotting AI-synthesized fake faces with the basic insight that the neuron coverage behaviors between real and fake faces in deep FR systems provide an important clue for differentiating them. Specifically, we adopt three deep FR systems (VGG-Face, OpenFace, and FaceNet) to capture the neuron coverage behavior of real and fake faces and train a simple linear classifier to detect them. In our work, the neuron activation is determined by our proposed neuron coverage criteria, named MNC, which specifies each layer a fixed threshold by learning from large real and fake face samples. Experiments carried on three deep FR systems, with two GAN variants for synthesizing fake faces, and on two high-resolution face dataset demonstrated the effectiveness of our proposed approach FakeSpotter. Our main contributions are summarized as follows.

  • In this paper, we observed that layer-by-layer neuron activation patterns in deep FR systems capture more subtle features that can be used in differentiating real and fake faces. Based on this observation, we propose a novel fake faces detection approach, named FakeSpotter, by monitoring neuron coverage behaviors in deep FR systems.

  • We implement the proposed approach and evaluate it on three DL based FR systems, with two variant GANs for synthesizing fake faces, and on two publicly face datasets. We also build a high-quality fake faces dataset for evaluating. Experimental results demonstrate the effectiveness of our approach.

  • Our proposed fake face detection method also provides a novel insight for detecting AI-synthesized fake video (e.g., face swapping), a type of well-known Deep Fakes.

The remainder of our paper is organized as follows. Section 2 discusses the related work. Section 3 presents our detecting approach in detail. We show the experimental results and demonstrate the effectiveness of our proposed approach in spotting fake faces in Section 4. Finally, Section 5 concludes this paper and discusses future research directions.

2 Related Work

In this section, we briefly summarize the literature on synthesizing fake images and detecting AI-synthesized fake faces.

2.1 Image Synthesis

GAN has achieved impressive progress in image synthesis which is the most well-studied area of the applications of GAN since it is first proposed in 2014 [8]. The generator in GAN learns to produce fake samples that are almost identical to real samples, while the discriminator learns to differentiate them.

Recent work in image synthesis like PGGAN [13], StyleGAN [14], and StarGAN [3] achieved remarkable results in face image synthesis. PGGAN [13] proposes a new training methodology for GANs with low-resolution images by growing both the generator and discriminator progressively. PGGAN can effectively produce images in large resolution with unprecedented quality. StyleGAN [14] redesigns the generator architecture by adding “style” of images at the convolution layers, which can transfer the collected styles in image synthesis.

These GANs can be well applied in synthesizing fake faces or editing facial attributes that are indistinguishable to humans. Thus, it is critical to develop effective fake image detection techniques and stop fakes spreading in case of social issues and protecting users’ privacy. In this paper, PGGAN and StyleGAN, created by NVIDIA, are adopted in generating fake faces for evaluating the effectiveness of our proposed method in detecting fake faces.

2.2 Fake Face Detection

Recently, some research [22, 20, 15, 40] borrow ideas from the traditional digital image forensics to detect GAN-synthesized fake faces/images. Several researchers [6, 11, 21] propose learning-based methods by modifying the architectures and loss functions of DNNs. The biggest challenge of these detection methods is that attackers can easily craft fake images by adding noises into images to bypass the fake face detection without effort.

Nataraj et al. [22] introduce a pixel-level fake images detection approach by computing the image co-occurrence matrics on RGB channels and training a DNN to learn the feature of co-occurrence matrics. McCloskey et al. [20] investigate that in generating images with GANs, the frequency of saturated pixels is limited as the generator’s internal values are normalized and the formation of a color image is different from a camera which is sensitive to spectral. Zhou et al. [40] adopt a faster R-CNN framework for detecting manipulated images by exploring RGB tampering artifacts and image noise feature, while this work focus on computer graphics-based tampering techniques not AI techniques. Adobe proposes a method to detect manipulated images with Photoshop which also manipulates images with computer graphics techniques [34]. These works are easily affected by adding noises into images. Mo et al. propose a CNN based method for fake face detection with an observation that some statistical artifact would be introduced in image synthesis. However, CNN based learning approach is vulnerable to adversarial examples.

Some research explore various ad hoc features for differentiating real and fake faces. Huh et al. [12] observed that the image EXIF metadata can be used to differentiate fake faces. However, the EXIF metadata can be easily modified or removed by attackers. Yang et al. [37] explore the difference between the locations of facial landmark points in real faces and GAN-synthesized faces and train a classifier to differentiate them, however, this difference can be mitigated with advanced GANs like StyleGAN and well-trained models. Therefore, some researchers investigate the unique fingerprints of GANs in image synthesis [39, 23] by detecting fake images in a general way. However, attacker could adopt an unseen GANs for generating fake images, which result the existing fingerprints of GANs failed in identifying fake faces. Our work also aims at proposing a general approach for detecting various GAN-synthesized fake faces.

Some studies [37, 1, 9] research Deep Fakes detection, mostly detecting face-swapping. These work mainly utilize the special characteristics in fake videos like the inconsistency of frames in a video, the unique facial movements or expressions, etc. However, these techniques in Deep Fakes detection cannot be applied directly in detecting fake images.

3 Our Method

In this section, we explain the detail of our proposed method in detecting fake faces generated with GANs, such as PGGAN and StyleGAN. Figure 2 presents an overview of our approach to detecting fake faces. Deep FR systems capture the neuron coverage behaviors of real and fake faces to train a simple linear binary-classifier for differentiating fake and real faces. Specifically, we first introduce the neuron coverage technique to capture the fine-grained features of faces with deep FR systems (e.g., VGG-Face, OpenFace, and FaceNet). Then, a neuron coverage criteria MNC is proposed for determining the neuron activation. Finally, we differentiate the fake and real faces with a linear binary-classifier based on the neuron coverage behaviors captured by deep FR systems.

Figure 2: An overview of the proposed FakeSpotter fake face detection method. Compared to the traditional learning based method (shown at the bottom), the FakeSpotter uses layer-wise neuron behavior as features, as opposed to final-layer neuron output.

3.1 Neuron Coverage

Neuron coverage is widely adopted in assuring the quality and security of DNNs and plays an important role in analyzing DNNs [36, 7, 16, 18, 17]. It motivated by code coverage by exploring the activate neurons whose output is higher than a specified threshold to investigate DNN internal behaviors. Recent research demonstrated that software errors/bugs and adversarial example attacks can be detected with neuron coverage techniques [25, 33, 35, 31, 19]. Next, we will introduce the computation of neuron output and present their powerful functionalities in representing input features.

In training a DNN model, neuron learns a particular representation of inputs with a complex nonlinear computation for learning models’ parameters. In prediction, a DNN model output the final result of an input by computing all the neurons output with learned parameters. In computing neuron output in DNNs, an activation function (e.g., sigmoid, ReLU, etc.) in DNNs returns the final result of a neuron’s output by receiving an input which is computed by connected neurons in the previous layer, weights matrix , and bias .

In image processing, pixels represent the raw features of physical images, while neurons, basic computing units in DNNs, learn the fine-grained features of images by the multiple layers in DNNs. We observed that neurons in DNNs are analogous to pixels in raw physical images and learn meaningful representations of inputs in a multi-view way. In this work, we hunt the difference between real and fake faces by capturing their neuron behaviors in deep FR systems, specifically the activated neurons.

Face recognition achieves significantly breakthrough since Facebook developed DeepFace [30] in 2014. DeepFace learns the face representation from a nine-layer neural network and achieves an accuracy of 97.53% on the evaluated dataset, Labeled Faces in the Wild (LFW). Some work are promoting the performance of DNNs in face recognition step-by-step with well-designed architectures and loss functions, like VGG-Face, OpenFace, and FaceNet, etc. These work even outperform humans in face recognition and verification. We leverage the distinguished performance of these deep FR systems in face representation to capture the neuron behaviors in learning facial features.

A DNN model includes multiple layers, and each layer contains numerous neurons. In this work, we pay attention to neurons in convolution and fully connected layers in deep FR systems, as the basic architecture of deep FR systems (e.g., VGG-Face, OpenFace, and FaceNet) is CNNs. The convolution layers perform convolution operations in DNNs and the fully connected layers combine the final results of previous layers in a CNN model. Both of these two types of layers learn representative representations of images in CNN models.

3.2 Neuron Activation in Deep FR Systems

In deep FR systems, neurons in different layers learn their specific facial representations. Given a trained deep FR model with fixed parameters (e.g., weights and bias), a neuron’s output is determined by the input face and always changes within a range. Here, we define neuron activation by deciding whether its output is larger than a specified threshold of .

In this paper, we propose a novel neuron coverage criteria MNC, by specifying a fixed threshold for each layer. The threshold is a mean value of all the neurons output in each layer. The mean value is calculated from large numbers of training samples including fake and real faces for determining whether a neuron is activated. Furthermore, the mean value may truly reflect the neuron coverage behaviors while simply specifying a fixed threshold is too rough for determining all the neurons’ activation in every layer like NC criteria [25]. We calculate the threshold for each layer by the following formula:


where is the total number of neurons in the th layer, = is a set of training inputs and indicates the number of training inputs in , denotes the th input in . Finally, the neuron activation with MNC for each layer is defined as follows:


where represents the input face, represents the th layer in DNN, represents neuron in layer , is a function to compute the neuron output, is the threshold of the th layer (see Equation (1)).

3.3 Detecting Fake Faces

We capture the neuron coverage behaviors with deep FR systems and train a linear binary-classifier for detecting fake faces. The number of activated neurons in each layer is formed as a feature vector when training a linear binary-classifier for differentiating fake and real faces.

Algorithm 1 describes the procedure of fake face detection. First, the threshold for determining neuron activation in each layer are identified by our proposed neuron coverage criteria with real and fake faces as training datasets, denoted as and . Then, a feature vector for each input face is formed as the number of activated neurons in each layer. Let and represent the feature vector of fake and real input faces respectively, where and are the number of activated neurons in the th and th layer, is the total number of layers in deep FR system. Finally, we train a supervised binary-classifier, denoted as , by receiving the formed feature vectors of fake and real faces as inputs to predict fake faces in the wild. In our work, we employ an support vector machine (SVM) model as our linear binary-classifier.

Input : Training dataset for fake and real faces , , Test dataset for fake and real faces , , Pre-trained deep FR model
Output : Binary-classifier , Prediction score , Label
1 ;
2 Determining the threshold of neuron activation for each layer.
3 for  do
4        for ,  do
5               ;
6               for  do
7                      ,
8       ,
9 Training a binary-classifier for identifying fake faces.
10 for ,  do
11        for ,  do
12               if  then
14              if  then
16 for  do
return , , ,
Algorithm 1 Algorithm for detecting fake faces with neuron coverage in deep FR systems

In prediction, an input face should be processed by a deep FR system to extract the neuron coverage behaviors with criteria MNC, actually the activated neurons in each layer. The activated neurons are formed as a feature to represent the input face. Then, the trained binary-classifier predicts whether the input is a fake face.

4 Experiments

In this section, we first introduce the experiment settings for evaluating the effectiveness of our method including basic dataset, techniques for face synthesis, and deep FR systems adopted for capturing activated neurons. Then, we present the experiment results by demonstrating the effectiveness in detecting fake faces, comparing with binary-classification method, and the robustness to potential attacks in adversarial settings. Finally, we discuss the limitations of our approach in identifying fake face images and the potential in fake videos detection.

4.1 Experiment Settings

4.1.1 Dataset:

We select two publicly popular human faces datasets created by NVIDIA as our real faces dataset, i.e., CelebFaces Attributes High Quality Database (CelebA-HQ)444https://github.com/tkarras/progressive_growing_of_gans and Flicker-Faces-HQ (FFHQ)555https://github.com/NVlabs/ffhq-dataset. CelebA-HQ has 30,000 faces, while FFHQ has 70,000 faces. Both of them offer high-resolution face images at . We created fake faces dataset with current state-of-the-art GANs in faces synthesis due to the unavailable of high-quality AI-synthesized fake faces dataset.

4.1.2 Fake Faces Synthesis:

In our work, we focus on GAN-synthesized fake faces detection. Recent work FaceForensics++ [26] provides a benchmark for facial manipulation detection. Faces in FaceForensics++ are manipulated by three methods, Deepfakes, Face2Face, and FaceSwap. The first method Deepfakes is learning-based by performing faces swapping and the last two methods are based on classical computer graphics techniques. Thus, faces in FaceForensics++ cannot match our requirements. Some apps (e.g., FaceApp, Reflect and ZAO, etc.) also provide the functionalities for synthesizing fake faces with GANs, but the quality of these synthesized faces is really poor.

In our experiment, we created a high-quality fake faces dataset with two GAN variants in image synthesis, PGGAN, and StyleGAN, for evaluating our detection approach on powerful GPU servers ( four NVIDIA Tesla P40 GPUs). InterFaceGAN [28] provides a simple way for us to edit facial attributes with PGGAN and StyleGAN. StyleGAN Encoder666https://github.com/Puzer/stylegan-encoder allows us to control the faces smiling in a fine-grained way. In this paper, we implement the fake faces dataset created with InterFaceGAN777https://github.com/ShenYujun/InterFaceGAN and StyleGAN Encoder.

4.1.3 Deep FR Systems:

Various FR models are developed and achieved remarkable performance in face recognition since Facebook developed DeepFace in 2014. These models are optimized with architectures or loss functions redesigned. In this paper, we select VGG-Face [24], OpenFace [2], and FaceNet [27] to leverage their powerful capabilities in face representation by capturing the activated neurons for differentiating fake and real faces.

Other experiments except the fake faces synthesis are performed on a server running Ubuntu 16.04 system on an 18-core 2.30GHz Xeon CPU with 200GB RAM and an NVIDIA Tesla M40 GPU with 24GB memory.

4.2 Experimental Results

We evaluated the effectiveness of our proposed approach in spotting fake faces in three ways. We assessed the performance of our approach in detecting fake faces by comparing with DL based binary-classifiers and discuss the robustness of our approach to potential attacks in adversarial settings.

4.2.1 Performance

Deep learning models have achieved tremendous progress in image classification due to the well-designed architecture, especially the deeper layers and powerful computing resources. Thus, we explore whether we can leverage the power of current DL models in image classification to identify fake faces.

In our experiment, we select three well-performed CNN models in image classification, namely VGG19 [29], ResNet50 [10], and Xception V1 [4], to build three binary-classifiers for identifying fake faces. The binary-classification models are trained on our collected real faces and generated fake faces dataset. Table 1 shows us the statistical data of training and test dataset. We randomly select real faces from the two well-known faces dataset FFHQ and CelebA-HQ, while the fake faces are generated with StyleGAN and PGGAN with the provided pre-trained model. We implement the three CNN models on Keras 2.2.4 with TensorFlow 1.12 backend.

Dataset Real Faces Fake Faces
Training 2,500 2,500 3,000 1,000
Test 500 500 400 600
Total 6,000 5,000
Table 1: The statistical data of training and test dataset for evaluating the accuracy of binary-classifiers in detecting AI-synthesized fake faces.

Experimental results show that VGG19, ResNet50, and Xception V1 give an accuracy of 69.95%, 70.11%, and 58.92% on the test dataset respectively. Obviously, the performance of DL based binary-classifier in fake faces detection is not acceptable in practice for its low detection accuracy. Next, we present the performance of our proposed approach in detecting fake faces with deep FR systems.

We first describe the structure information of the three adopted deep FR systems and the training and test dataset in our experiments, then present the experimental results. Table 2 shows us the basic information of the three adopted deep FR systems in our experiments, where neuron, layers_total, and parameters mean the total number of neurons, layers, and parameters in the system respectively, architecture denotes the CNN model adopted in the system, and layers_select indicates the number of selected layers in deep FR systems for capturing neuron output, specifically the convolution and fully connected layers.

Deep FR
VGG-Face OpenFace FaceNet
Neurons 63K 39K 163K
Layers_total 26 157 426
Layers_select 19 38 48
Parameters 145M 3M 22M
Architecture VGGNet16 GoogleNet GoogleNet
Table 2: The description of three adopted deep FR systems, VGG-Face, OpenFace, and FaceNet.

Table 3 shows us the collected real faces and generated fake faces in our experiments for evaluating the performance of our proposed approach. In generating fake faces, we select two GANs to generate fake faces. StyleGAN generates fake faces with smiling expressions (see the second row faces in Figure 1), while PGGAN generates fake faces with different poses (see the first row faces in Figure 1) and smiling expressions. For each facial expression, we generate five synthesized fake faces for one person.

Dataset Real Faces Fake Faces
Training 5,000 3,000 2,000
Test 1,000 200 3,00
Total 6,000 5,500
Table 3: Dataset for evaluating the performance of our neuron coverage-based approach in detecting AI-synthesized fake faces.

Figure 3 visualizes the number of activated neurons of four randomly selected synthesized fake faces in deep FR systems. Neurons activation is determined by our proposed neuron coverage criteria MNC. In Figure 3, the synthesized fake face are from one person and fake face are from another person. Figure 3(a) presents the activated neurons of four fake faces in VGG-Face, which selects VGGNet16 as the basic architecture. Figure 3(b) shows us the activated neurons in FaceNet, which selects GoogleNet as the basic architecture. Both OpenFace and FaceNet select GoogleNet as their basic architecture and OpenFace is based on FaceNet. Therefore, we present FaceNet to represent the capabilities of GoogleNet in capturing neuron coverage behaviors with MNC. Figure 3 tells us that different fake faces share strong similarities in neuron coverage behaviors with our proposed criteria MNC.

(a) Neuron activation in VGG-Face
(b) Neuron activation in FaceNet
Figure 3: The distribution of activated neurons in deep FR systems. Horizontal axis denotes the index of layers that contain activated neurons, while the vertical axis denotes the number of activated neurons in each layer.

Experimental results show that our neuron coverage behaviors based approach give an accuracy of 78.23%, 80.54%, 84.78% on VGG-Face, OpenFace, and FaceNet in detecting fake faces respectively. In comparing with the deeply trained CNNs, our approach performs better results in detecting fake faces in the wild. Table 4 and Figure 4 present the experimental results between our approach and deep CNN based approach in detecting fake faces.

Figure 4: The accuracy of Deep CNN model and our neuron coverage behaviors based approach in spotting fake faces.The first three approaches utilize deep CNN model (e.g, Xception V1, VGG19, and ResNet50, etc.) to detect fake faces. The last three approaches leverage neuron coverage in face recognition systems (e.g., VGG-Face, OpenFace, and FaceNet, etc.) to spot fake faces.
Approach Deep CNN Model Neuron Coverage-based
Model Xception V1 VGG19 ResNet50 VGG-Face OpenFace FaceNet
Accuracy 58.92% 69.95% 70.11% 78.23% 80.54% 84.78%
Table 4: Performance between deep CNN model and our neuron coverage behaviors based approach in spotting fake faces. The second column is three typical deep CNN model (e.g., Xception V1, VGG19, and ResNet50, etc.) which achieved excellent results in image classification, while the third column is three state-of-the-art deep face recognition systems which performs well in face recognition and verification.VGG-Face adopts VGG16, while OpenFace and FaceNet employ GoogleNet as an alternative architecture.

4.2.2 Robustness to Attacks in Adversarial Settings

Our neuron coverage based fake faces detection approach differentiate the real and fake faces by capturing their neurons’ behaviors in deep FR systems. Neuron learns the fine-grained features of input faces by activating specific neurons in each layer. Adding noises such as modifying some pixels in images will keep the images’ semantics unchanged. Thus, neuron behaviors would remain unchanged.

Adversarial example modifies pixels in photos intentionally would not change the faces’ content and keep the photos’ semantics unchanged, while it may change some neurons output and finally cause incorrect prediction results. Adding noises into photos by modifying pixels randomly will also change some neurons output, but both the final prediction results and the photos’ semantic features are unchanged. These two attacks cause some neurons output changed, while our approach captures all the activated neurons in every layer as a feature for spotting fake faces. Thus, our approach can be hardly suffered by these attacks.

4.3 Discussion

Our approach achieved attractive results than deeply trained CNNs in spotting AI-synthesized fake faces. However, there are also some limitations of our proposed approach. The first one is that attackers will consider the neuron coverage in synthesizing fake faces by analyzing the existing deep FR systems. But it’s a tough task, which requires tuning parameters and keeps the faces realistically. The second one is that capturing neuron outputs costs too much time, especially the FaceNet [27]. We will investigate the highly valuable neurons and capture these neurons output for differentiating fake faces to reduce computing costs significantly.

Our approach also shows potential in detecting fake videos, like faces swapping. To the best of our knowledge, the techniques of existing faces swapping desktop software and mobile apps are mostly based on GANs. In the fake videos, original faces are replaced with the synthesized faces in each frame and then a series of frames are combined for generating video. Thus, each frame in the fake video is still an image that is synthesized by GANs. Our proposed approach identities the fake video by detecting the synthesized fake faces images in the video frame-by-frame.

5 Conclusion and Future Research Directions

In this paper, we present a neuron coverage based approach for detecting AI-synthesized fake faces. We explore the power of neurons in deep FR systems in representing fine-grained features of faces. In deep FR systems, we investigate the capabilities of activated neurons in representing face images for differentiating real and fake faces. We evaluate our approach on our generated fake faces dataset with two GANs and three deep FR systems. Experiment results demonstrate the effectiveness of our approach in spotting fake faces. In particular, our approach can be applied in Deep Fakes detection, for instance, faces swapping. In our future work, we will seek a practical way to localize the tampered region in the fake faces as it provides strong evidence for us in forensics and track, in addition, we will also build a benchmark including high-quality fake images, tools for synthesizing fake images, and existing detection methods to advance this emerging field.

With the rapid development of AI techniques, realistic fake multimedia (e.g., fake speech, fake videos, etc.) is becoming a rising threat to common users, celebrities, and even politicians in the future. The arms race between creating fakes and fighting fakes is on the endless road, and powerful weapons should be developed for protecting the privacy of us while we are enjoying the AI techniques.


  • [1] S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li (2019) Protecting world leaders against deep fakes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 38–45. Cited by: §2.2.
  • [2] B. Amos, B. Ludwiczuk, and M. Satyanarayanan (2016) OpenFace: a general-purpose face recognition library with mobile applications. Technical report CMU-CS-16-118, CMU School of Computer Science. Cited by: §1, §4.1.3.
  • [3] Y. Choi, M. Choi, M. Kim, J. Ha, S. Kim, and J. Choo (2018) StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789–8797. Cited by: §1, §2.1.
  • [4] F. Chollet (2017) Xception: deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258. Cited by: §4.2.1.
  • [5] S. Cole (2018) We Are Truly F—ed: Everyone Is Making AI-Generated Fake Porn Now. Note: https://www.vice.com/en_us/article/bjye8a/reddit-fake-porn-app-daisy-ridley/(Jan 25 2018) Cited by: §1.
  • [6] L. Dang, S. Hassan, S. Im, J. Lee, S. Lee, and H. Moon (2018) Deep learning based computer generated face identification using convolutional neural network. Applied Sciences 8 (12), pp. 2610. Cited by: §1, §2.2.
  • [7] X. Du, X. Xie, Y. Li, L. Ma, Y. Liu, and J. Zhao (2019) DeepStellar: model-based quantitative analysis of stateful deep learning systems. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 477–487. Cited by: §3.1.
  • [8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680. Cited by: §2.1.
  • [9] D. Güera and E. J. Delp (2018) Deepfake video detection using recurrent neural networks. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6. Cited by: §2.2.
  • [10] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §4.2.1.
  • [11] C. Hsu, C. Lee, and Y. Zhuang (2018) Learning to detect fake face images in the wild. In 2018 International Symposium on Computer, Consumer and Control (IS3C), pp. 388–391. Cited by: §1, §2.2.
  • [12] M. Huh, A. Liu, A. Owens, and A. A. Efros (2018) Fighting fake news: image splice detection via learned self-consistency. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 101–117. Cited by: §1, §2.2.
  • [13] T. Karras, T. Aila, S. Laine, and J. Lehtinen (2017) Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. Cited by: Figure 1, §1, §2.1.
  • [14] T. Karras, S. Laine, and T. Aila (2019) A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410. Cited by: Figure 1, §1, §2.1.
  • [15] H. Li, B. Li, S. Tan, and J. Huang (2018) Detection of deep network generated images using disparities in color components. arXiv preprint arXiv:1808.07276. Cited by: §1, §2.2.
  • [16] L. Ma, F. Juefei-Xu, M. Xue, B. Li, L. Li, Y. Liu, and J. Zhao (2019) DeepCT: tomographic combinatorial testing for deep learning systems. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 614–618. Cited by: §3.1.
  • [17] L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, et al. (2018) DeepGauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 120–131. Cited by: §3.1.
  • [18] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao, et al. (2018) Deepmutation: mutation testing of deep learning systems. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), pp. 100–111. Cited by: §3.1.
  • [19] S. Ma, Y. Liu, G. Tao, W. Lee, and X. Zhang (2019) NIC: detecting adversarial samples with neural network invariant checking.. In Proceedings of the Network and Distributed System Security Symposium, Cited by: §3.1.
  • [20] S. McCloskey and M. Albright (2018) Detecting GAN-generated imagery using color cues. arXiv preprint arXiv:1812.08247. Cited by: §1, §2.2, §2.2.
  • [21] H. Mo, B. Chen, and W. Luo (2018) Fake faces identification via convolutional neural network. In Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia Security, pp. 43–47. Cited by: §1, §2.2.
  • [22] L. Nataraj, T. M. Mohammed, B. Manjunath, S. Chandrasekaran, A. Flenner, J. H. Bappy, and A. K. Roy-Chowdhury (2019) Detecting GAN generated fake images using co-occurrence matrices. arXiv preprint arXiv:1903.06836. Cited by: §1, §2.2, §2.2.
  • [23] Y. Ning, D. Larry, and F. Mario (2018) Attributing fake images to gans: learning and analyzing gan fingerprints. arXiv preprint arXiv:1811.08180v3. Cited by: §2.2.
  • [24] O. M. Parkhi, A. Vedaldi, A. Zisserman, et al. (2015) Deep face recognition.. In bmvc, Vol. 1, pp. 6. Cited by: §1, §4.1.3.
  • [25] K. Pei, Y. Cao, J. Yang, and S. Jana (2017) DeepXplore: automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, pp. 1–18. Cited by: §3.1, §3.2.
  • [26] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner (2019) FaceForensics++: learning to detect manipulated facial images. In International Conference on Computer Vision (ICCV), Cited by: §4.1.2.
  • [27] F. Schroff, D. Kalenichenko, and J. Philbin (2015) Facenet: a unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823. Cited by: §1, §4.1.3, §4.3.
  • [28] Y. Shen (2019) Interpreting the latent space of gans for semantic face editing. arXiv preprint arXiv:1907.10786. Cited by: §4.1.2.
  • [29] K. Simonyan and A. Zisserman (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Cited by: §4.2.1.
  • [30] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf (2014) Deepface: closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701–1708. Cited by: §3.1.
  • [31] G. Tao, S. Ma, Y. Liu, and X. Zhang (2018) Attacks meet interpretability: attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems, pp. 7717–7728. Cited by: §3.1.
  • [32] S. Tariq, S. Lee, H. Kim, Y. Shin, and S. S. Woo (2019) GAN is a friend or foe?: a framework to detect various fake face images. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pp. 1296–1303. Cited by: §1.
  • [33] Y. Tian, K. Pei, S. Jana, and B. Ray (2018) DeepTest: automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering, pp. 303–314. Cited by: §3.1.
  • [34] S. Wang, O. Wang, A. Owens, R. Zhang, and A. A. Efros (2019) Detecting photoshopped faces by scripting photoshop. arXiv preprint arXiv:1906.05856. Cited by: §2.2.
  • [35] X. Xie, L. Ma, F. Juefei-Xu, M. Xue, H. Chen, Y. Liu, J. Zhao, B. Li, J. Yin, and S. See (2019) DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 146–157. Cited by: §3.1.
  • [36] X. Xie, L. Ma, H. Wang, Y. Li, Y. Liu, and X. Li (2019) DiffChaser: detecting disagreements for deep neural networks. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Cited by: §3.1.
  • [37] X. Yang, Y. Li, H. Qi, and S. Lyu (2019) Exposing gan-synthesized faces using landmark locations. arXiv preprint arXiv:1904.00167. Cited by: §2.2, §2.2.
  • [38] E. Zakharov, A. Shysheya, E. Burkov, and V. Lempitsky (2019) Few-shot adversarial learning of realistic neural talking head models. arXiv preprint arXiv:1905.08233. Cited by: §1.
  • [39] X. Zhang, S. Karaman, and S. Chang (2019) Detecting and simulating artifacts in gan fake images. arXiv preprint arXiv:1907.06515. Cited by: §2.2.
  • [40] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis (2018) Learning rich features for image manipulation detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1053–1061. Cited by: §2.2, §2.2.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description