A novel robust active appearance model (AAM) matching algorithm is presented. The method consists of two main stages. First, initial residuals are clustered by a non parametric mean shift mode detection step. Second, modes without gross outliers are selected using an objective function. Robustness of the matching procedure is demonstrated on a variety of examples with different noise conditions. The proposed algorithm outperformed the conventional AAM matching on images with gross disturbances and can tolerate up to 40% of disturbed data.