APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment
Audio-guided face reenactment aims to generate a photorealistic face that has matched facial expression with the input audio. However, current methods can only reenact a special person once the model is trained or need extra operations such as 3D rendering and image post-fusion on the premise of generating vivid faces. To solve the above challenge, we propose a novel Real-time Audio-guided Multi-face reenactment approach named APB2FaceV2, which can reenact different target faces among multiple persons with corresponding reference face and drive audio signal as inputs. Enabling the model to be trained end-to-end and have a faster speed, we design a novel module named Adaptive Convolution (AdaConv) to infuse audio information into the network, as well as adopt a lightweight network as our backbone so that the network can run in real time on CPU and GPU. Comparison experiments prove the superiority of our approach than existing state-of-the-art methods, and further experiments demonstrate that our method is efficient and flexible for practical applications
Jiangning Zhang, Xianfang Zeng, Chao Xu, Jun Chen, Yong Liu, Yunliang Jiang
\addressZhejiang University, Hangzhou, Zhejiang, China
Huzhou University, Huzhou, Zhejiang, China
audio-guided generation, multi-face reenactment, adaptive convolution, generative adversarial nets
Audio-guided face reenactment is a task to generate photorealistic face images under the condition of audio input, which has promising applications such as animation production, virtual announcer, and game. In this paper, different from current methods that can only reenact a special person once the model is trained, we focus on a more challenging task: audio-guided multi-face reenactment, where different target faces among several persons can be reenacted using one unified model.
Benefited from the development of neural network, many methods have achieved good results in the audio-to-face task. Cudeiro et al.  present a speech driven facial animation framework named VOCA that can fully automatically outputs a realistic character animation given a speech signal and a static character mesh. Works [2, 3] employ the LSTM  model to generate orofacial movement from acoustic features for predefined 3D model. Though 3D model-based methods can obtain vivid results, they need high costs for hardware and predefined 3D model as well as post-processing time consumption. Thus the pixel-based method is born to conduct the audio-to-face task. Duarte et al.  propose the Wav2Pix to generate the face image by an encoded audio vector in an adversarial manner, while Zhang et al.  design an APB2Face model that immensely improves the quality of the generated image. Consistent with pixel-based method, we design our model in a generative adversarial manner that can reenact photorealistic images and is easy to follow.
However, almost all of the current approaches [7, 5, 6] can only reenact one special person once the model is trained on the premise of generating vivid faces, meaning they are not competent to reenact various faces using a unified model. In order to solve the above problem, we propose a novel APB2FaceV2 to reenact different target faces among multiple persons with corresponding reference face and drive audio information, which has more practical application value. Specifically, the proposed APB2FaceV2 consists of an Audio-aware Fuser that extracts embedded geometric vector from input audio, head pose, and eye blink information, as well as a Multi-face Reenactor that generates target faces with a reference face and the geometric vector. At the same time, we find that nearly all current approaches do not take the model size into account that is important for practical applications. So we come up an Adaptive Convolution (AdaConv) to infuse audio information into the network so that the model can be trained in an end-to-end manner, as well as employ a modified lightweight network  as our backbone so that the model can run in real time. Specifically, we make the following four contributions:
i) A novel APB2FaceV2 is proposed to to reenact different target faces among multiple persons using one unified model.
ii) We design a new vector-based information injection module named AdaConv that achieves an end-to-end training.
iii) A lightweight backbone is adopted so that the method can run on CPU or GPU in real time.
iv) Experimental results demonstrate the efficiency and flexibility of our proposed approach.
2 Related Works
Generative Adversarial Networks. Since the concept of generative adversarial network was first proposed , many excellent works have been proposed to generate photorealistic images. Generally, these methods mainly fall into two categories: the vector-based method [10, 11, 12] that only uses a noise or embedded vector as input to generate target image, and the pixel-based method [13, 14] that uses the image as input. Theoretically, each of these methods contains a generator G with parameter to capture the data distribution for generating photorealistic images, as well as a discriminator D to authenticate generated images for enhancing the capability of G in an adversarial manner. To learn the distribution of G over data from a prior distribution (), D plays a two-player minimax game with G in the following value function :
Our proposed method belongs to the pixel-based category that inputs an image instead of only a vector, which is more efficient and practical than vector-based method.
Face Reenactment via Audio. Many works have yielded good results in the audio-to-face task, which uses audio as input for providing adequate information about orofacial movements. Works [15, 2, 1, 3] use the audio signal to predict parameters of the predefined 3D model, while Suwajanakorn et al.  and Prajwal et al.  propose to predict the lip rather the full face. Thus these methods need extra post-operations such as 3D rendering or face fusion, which is cumbersome and not suitable for practical applications. We wish design an end-to-end method to directly generate the full face like [18, 19, 7, 20, 21], and supplement some auxiliary signals simultaneously to control the facial areas that are not related to the audio information. So based on the previous work , we design a new framework named APB2FaceV2 that can not only generate photorealistic face end-to-end but also reenact multiple faces in real time by a unified model.
As depicted in Figure 1, we propose a novel APB2FaceV2 framework, which consists of an Audio-aware Fuser () and a Multi-face Reenactor (), to complete a more challengeable audio-guided multi-face reenactment task efficiently in real time. Detailed implementation and source code are available.
Audio-aware Fuser. The Audio-aware Fuser module inputs audio, head pose, and eye blink signals, which are further extracted by , , and to obtain , , and respectively. Specifically, contains 5 convolutional layers for extracting the feature of each time node and additional 5 convolutional layers for fusing them, while both modules and contain three linear layers. Subsequently, the three features are concatenated and further extracted through to obtain the embedded representation of the facial geometric feature , where the facial landmark is used as the supervisory signal in the training stage.
Multi-face Reenactor. Given a reference face image and the extracted facial geometric feature , the Multi-face Reenactor reenacts the target face that has matched facial expression with the input signals, i.e. audio, head pose, and eye blink. Specifically, consists of a chain of sub-modules: an image encoder , a feature transformer ( represents the number of module repetitions and is set to 9 in the paper), and an image decoder . The process can be described as:
Note that the feature transformer is designed in a decoupling idea that simultaneously learns appearance information from as well as the geometric information from , and the new proposed AdaConv is used to inject geometric information on each block.
Adaptive Convolution. Different from APB2Face  that injects facial movement information by first plotting the landmark image and then concatenating it with deep features, we propose an elegant information injection module, i.e. Adaptive Convolution (AdaConv). As shown in the top right of Figure 1 (Highlighted in green), the AdaConv layer inputs a geometric vector and a deep feature map , and outputs a modified feature map . In detail, goes through two linear layers to generate a set of parameters, i.e. , which will be reshaped and applied to a convolutional layer. As the formula shows below:
The parameter of the convolutional layer contains weight parameters and bias parameters, where , , and are kernel size, channel number, and group number for the convolution. Thus we can control the amount of injected information by controlling the values of these parameters. Specifically, AdaConv reduces to AdaIN  when we set , and we set in the paper for balancing computation and model effect.
Objective Function. In the training stage, we adopt geometry and content losses to supervise geometric information and generated image quality, as well as adversarial loss to further improve the quality and authenticity of the reenacted image. The overall loss function is:
where , , and represent weight parameters to balance different terms, and are set 1, 100, and 1 respectively.
i) Geometry loss calculates error between the predicted facial geometric feature and corresponding real facial landmark .
ii) Content loss calculates error between the reenacted target face and corresponding real face .
iii) Adversarial loss adopts an extra discriminator to form a adversarial training against the reenactor that greatly improves the quality of the generated image.
where and stand distributions for real and generated fake images respectively.
Dataset. In the paper, almost all of experiments are conducted on AnnVI dataset that contains six announcers (three men and three women) and 23790 frames totally with corresponding audio clip, head pose, eye blink, and landmark .
Implementation Details. We use Adam optimizer  with , and train the model for 110 epoch. The learning rate is set to , and the batch size is 16. PatchGAN  is used as the discriminator, and the training setting is in accord with the reenactor.
Qualitative Results. Some qualitative experiments are conducted on AnnVI dataset to visually demonstrate the high quality of reenacted images and the flexibility of the proposed approach. Specifically, we randomly select one face of each identity (6 faces totally) as the reference face, and one drive frame of each identity for supplying audio, head pose, and eye blink signals. As indicated by the red rectangles in Figure 2, our proposed method can reenact photorealistic faces among multiple persons with one unified model that achieves multi-face reenactment task. Thanks to the decoupling design of our method, APB2FaceV2 can use input signals of other persons to reenact the target face that is consistent with the identity of the reference face. Experimental results show that our method has strong generalization ability, where the model can use non-self audio as input to reenact photorealistic faces.
Quantitative Results. As shown in Table 1, SSIM metric is chosen to quantitatively evaluate our method with the state-of-the-art (SOTA) method, and experimental results indicate that the proposed method can generate more photorealistic faces where the SSIM score goes up from 0.799 to 0.805, even though using only one unified model (The work  need train 6 models totally for 6 persons). Nevertheless, our method still obtains a higher Detection Rate (DR), i.e. 99.1%, which also demonstrates the superiority of our method than SOTA.
|Method||SSIM||FID||Params (M)||FPS (CPU)||FPS (GPU)|
Comparison with SOTAs. We further conduct a comparison experiment with most related SOTA methods on the Youtubers dataset . As show in Figure 3, our approach obtains a better visual effect than others, as well as the best SSIM and FID scores. Moreover, our method significantly reduces the number of parameters by nearly 6 times and 4 times compared to Wav2Pix and APB2Face respectively, and can run in real time, i.e. 22.5 FPS in CPU (i7-8700K @ 3.70GHz) and 158.9 FPS in GPU (2080 Ti), as shown in Table 2.
Decoupling Experiment. A decoupling experiment is further conducted to demonstrate that our proposed method is capable of disentangling input signals, i.e. audio, head pose, and eye blink. As shown in Figure 4, the first three rows are generated results that only change one component of the head pose signal, i.e. yaw, pitch, or roll, while the last row shows the results that only change the eye blink signal. Experimental results indicate that our method can control the properties of the generated face that is flexible for practical applications.
In this paper, we propose a novel APB2FaceV2 to address a more challengeable audio-guide multi-face reenactment task, which aims at using one unified model to reenact different target faces among multiple persons with corresponding reference face and drive audio signal as inputs. Specifically, an Audio-aware Fuser is firstly used to predict a geometric representation from input signals, and then Multi-face Reenactor fuse it with the reference face that supplies appearance information to reenact photorealistic target face. Besides, a novel AdaConv module is proposed to inject geometric information in a more elegant and efficient way. Extensive experiments demonstrate the efficiency and flexibility of our approach.
We will further combine Neural Architecture Search (NAS) with our approach to search for a more accurate and faster model for better practical applications, and we hope our work will help users to achieve better jobs.
- Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black, “Capture, learning, and synthesis of 3d speaking styles,” in CVPR, 2019.
- Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu, “Audio-driven talking face video generation with natural head pose,” arXiv preprint arXiv:2002.10137, 2020.
- Guanzhong Tian, Yi Yuan, and Yong Liu, “Audio2face: Generating speech/face animation from single audio with attention-based bidirectional lstm networks,” in ICMEW, 2019.
- Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber, “Lstm: A search space odyssey,” IEEE transactions on neural networks and learning systems, vol. 28, no. 10, pp. 2222–2232, 2016.
- Amanda Duarte, Francisco Roldan, Miquel Tubau, Janna Escur, Santiago Pascual, Amaia Salvador, Eva Mohedano, Kevin McGuinness, Jordi Torres, and Xavier Giro-i Nieto, “Wav2pix: Speech-conditioned face generation using generative adversarial networks.,” in ICASSP, 2019.
- Jiangning Zhang, Liang Liu, Zhucun Xue, and Yong Liu, “Apb2face: Audio-guided face reenactment with auxiliary pose and blink signals,” in ICASSP, 2020.
- Joon Son Chung, Amir Jamaludin, and Andrew Zisserman, “You said that?,” arXiv preprint arXiv:1705.02966, 2017.
- Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang, “Autogan-distiller: Searching to compress generative adversarial networks,” arXiv preprint arXiv:2006.08198, 2020.
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Generative adversarial nets,” in NeurIPS, 2014.
- Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” in ICLR, 2018.
- Tero Karras, Samuli Laine, and Timo Aila, “A style-based generator architecture for generative adversarial networks,” in CVPR, 2019.
- Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, “Analyzing and improving the image quality of stylegan,” in CVPR, 2020.
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros, “Image-to-image translation with conditional adversarial networks,” in CVPR, 2017.
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in ICCV, 2017.
- Najmeh Sadoughi and Carlos Busso, “Speech-driven expressive talking lips with conditional sequential generative adversarial networks,” IEEE Transactions on Affective Computing, 2019.
- Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman, “Synthesizing obama: learning lip sync from audio,” in ACM TOG, 2017.
- KR Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” in ACM MM, 2020.
- Olivia Wiles, A Sophia Koepke, and Andrew Zisserman, “X2face: A network for controlling face generation using images, audio, and pose codes,” in ECCV, 2018.
- Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T Freeman, Michael Rubinstein, and Wojciech Matusik, “Speech2face: Learning the face behind a voice,” in CVPR, 2019.
- Yeqi Bai, Tao Ma, Lipo Wang, and Zhenjie Zhang, “Speech fusion to face: Bridging the gap between human’s vocal characteristics and facial imaging,” arXiv preprint arXiv:2006.05888, 2020.
- Hyeong-Seok Choi, Changdae Park, and Kyogu Lee, “From inference to generation: End-to-end fully self-supervised generation of human face from speech,” 2020.
- Xun Huang and Serge Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in ICCV, 2017.
- Face++, ,” https://www.faceplusplus.com/attributes/, 2020, Accessed September 16, 2020.
- Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.