In this work a new robust color and contour based object detection method in images with varying shadows is presented. The method relies on a physics-based contour detector that emphasizes material changes and a contourbased boosted classifier. The method has been tested in a sequence of outdoor color images presenting varying shadows using two classifiers, one that learnt contour object features from a simple gradient detector, and another that learnt from the photometric invariant contour detector. It is shown that the detection performance of the classifier trained with the photometric invariant detector is significantly higher than that of the classifier trained with gradient detector. Key words: color invariance, shadow removal, object detection, boosting