The aim of this study is to propose an algorithm for simplifying color images. The assumption is made that a set of reference colors is available and that this set indicates which features are of importance in the image. The goal is then to simplify as much as possible the color image while preserving as much as possible the shapes that are similar to the reference set. The proposed algorithm is in two steps. In a first step, a scalar similarity image of same morphological structure as the color image is computed. This allows to associate similarity-level sets with the color image that are used, in the second step, for performing a similarity-adaptive simplification of the color image: each similarity-level set is associated with a certain filter, the filtering parameter being a decreasing function of the similarity.