Speech-Driven Face Synthesis from 3D Video

This paper presents a framework for speech-driven synthesis of real faces from a corpus of 3D video of a person speaking. Video-rate capture of dynamic 3D face shape and colour appearance provides the basis for a visual speech synthesis model. A displacement map representation combines face shape and colour into a 3D video. This representation is used to efficiently register and integrate shape and colour information captured from multiple views. To allow visual speech synthesis viseme primitives are identified from the corpus using automatic speech recognition. A novel non-rigid alignment algorithm is introduced to estimate dense correspondence between 3D face shape and appearance for different visemes. The registered displacement map representation together with a novel optical flow optimisation using both shape and colour, enables accurate and efficient non-rigid alignment. Face synthesis from speech is performed by concatenation of the corresponding viseme sequence using the non-r...

• Type: Conference
• Year: 2004
• Where: 3DPVT
• Authors: Ioannis A. Ypsilos, Adrian Hilton, Aseel Turkmani, Philip J. B. Jackson
• Keywords: 3DPVT 2004, Displacement Map Representation, Face Shape, Visual Speech Synthesis, Visualization