Finding faces in visually challenging environments is crucial to many applications, such as audio-visual automatic speech recognition, video indexing, person recognition, and video surveillance. In this study, we investigate several algorithms to improve face detection accuracy in visually challenging environments using the IBM appearance based face detection system. The algorithms considered are trainable skintone pre-screening, Hamming windowing of the face images, DCT coefficient selection, and the AdaBoost technique. When these methods are combined, an up to 68% relative reduction in face detection error is observed on visually challenging datasets.