Abstract. This paper proposes a new method to solve the problem of face recognition under varying illumination conditions. We introduce an exemplar-based technique to decouple and subsequently recover the canonical face and the illumination functions from the intensity images. The canonical image is equivalent to the reflectance field of the face that is invariant to illumination. We subsequently use the canonical face to synthesize novel face appearances together with a set of lighting models. We then demonstrate the ability of the synthesis approach to improve the performance of the face recognition task.