Face detection is a key problem in human-computer interaction. In this paper, we present an algorithm for rotation invariant face detection in color images of cluttered scenes. It is a hierarchical approach, which combines a skin color model, a neural network, and an upright face detector. Firstly, the skin color model is used to process the color image to segment the face-like regions from the background. Secondly, the neural network computing and an operation for locating irises are performed to acquire rotation angle of each input window in the face-like regions. Finally, we provide an upright face detector to determine whether or not the rotated window is a face. Those techniques are integrated into a face detection system. The experiments show that the algorithm is robust to different face sizes and various lighting conditions.