Avatar Puppetry Using Real-Time Audio and Video Analysis

Abstract. We present a system which consists of a lifelike agent animated in real-time using video and audio analysis from the user. This kind of system could be used for Instant Messaging where an avatar controlled like a puppet is displayed instead of the webcam flow. The overall system is made of video analysis based on Active Appearance Models and audio analysis based on Hidden Markov Model. The parameters from these two modules are sent to a control system driving the animation engine. The video analysis extracts the head orientation and the audio analysis provides the phonetic string used to move the lips. 1 System Overview

• Type: Conference
• Year: 2007
• Where: IVA
• Authors: Sylvain Le Gallou, Gaspard Breton, Renaud Séguier, Christophe Garcia
• Keywords: Audio Analysis, Hidden Markov Model, IVA 2007, Video Analysis