Abstract

This paper addresses the problem of face video inpainting. Existing video inpainting methods target primarily at natural scenes with repetitive patterns. They do not make use of any prior knowledge of the face to help retrieve correspondences for the corrupted face. They therefore only achieve sub-optimal results, particularly for faces under large pose and expression variations where face components appear very differently across frames. In this paper, we propose a two-stage deep learning method for face video inpainting. We employ 3DMM as our 3D face prior to transform a face between the image space and the UV (texture) space. In Stage I, we perform face inpainting in the UV space. This helps to largely remove the influence of face poses and expressions and makes the learning task much easier with well aligned face features. We introduce a frame-wise attention module to fully exploit correspondences in neighboring frames to assist the inpainting task. In Stage II, we transform the inpainted face regions back to the image space and perform face video refinement that inpaints any background regions not covered in Stage I and also refines the inpainted face regions. Extensive experiments have been carried out which show our method can significantly outperform methods based merely on 2D information, especially for faces under large pose and expression variations.

Method

Given a face video, we first fit 3DMM to every frame to obtain per-frame shape, texture, and pose parameters. The shape and pose parameters are used for transforming the face between the image space and UV space, while the texture parameters are used to generate synthesized texture to provide auxiliary information for the inpainting task. In Stage I, we first transform the face from the image space to the well-aligned UV space and use our proposed MUC-Net to perform UV-map completion. FA module is proposed for MUC-Net to facilitate the correspondence retrieval across UV texture frames. In Stage II, we transform the completed UV-map in Stage I back to the image space, and use our proposed FVR-Net to inpaint any background (non-face) regions not covered in Stage I as well as refine the inpainted face regions.

BibTeX

Acknowledgements

The website template was borrowed from Michaël Gharbi.