We present ongoing work on a project for automatic recognition of spontaneous facial actions. Spontaneous facial expressions differ substantially from posed expressions, similar to how spontaneous speech differs from directed speech. Previous methods for automatic facial expression recognition assumed images were collected in controlled environments in which the subjects deliberately faced the camera. Since people often nod or turn their heads, automatic recognition of spontaneous facial behavior requires methods for handling out-of-image-plane head rotations. There are many promising approaches to address the problem of out-of-image plane rotations. In this paper we explore an approach based on 3-D warping of images into canonical views. A front-end system was developed that jointly estimates camera parameters, head geometry and 3D head pose across entire sequences of video images. First a small set of images was used to estimate camera parameters and 3D face geometry. Markov chain M...