Face tracking in realistic environments is a difficult problem due to pose variations, occlusions of objects, illumination changes and cluttered background, among others. The paper presents a robust and real-time face tracking algorithm. A novel likelihood is developed based on a boosted multi-view face detector to characterize the structure information. The likelihood function is further integrated with particle filter which can maintain multiple hypotheses. The algorithm proposed is able to track faces in different poses, and is robust to temporary occlusions, illumination changes and complex background. In addition, it enjoys a real-time implementation. Experiments with a challenging image sequence shows the effectiveness of the algorithm.