Abstract. In this paper, we present an automatic human portrait system based on the And-Or graph representation. The system can automatically generate a set of life-like portraits in different styles from a frontal face image. The system includes three subsystems, each of which models hair, face and collar respectively. The face subsystem can be further decomposed into face components: eyebrows, eyes, nose, mouth, and face contour. Each component has a number of distinct sub-templates as a leaf-node in the And-Or graph for portrait. The And-Or graph for portrait is like a "mother template" which produces a large set of valid portrait configurations, which is a "composite templates" made of a set of sub-templates. Our approach has three novel aspects:(1) we present an And-Or graph for portrait that explains the hierarchical structure and variability of portrait and apply it into practice; (2) we combine hair, face and collar into a system that solves a practical prob...