We describe a subsystem of a content-based image retrieval (CBIR) environment that supports a user in the definition of image similarity. Out of a single image or a set of query images we refine a query model: a list of feature extraction functions with associated thresholds and weights. The subsystem aims at bridging the gap between a user’s high-level concepts and the low-level visual features employed and at supporting both, the casual user and the expert. The paper investigates and evaluates several approaches for this purpose within a CBIR system for coats of arms. A user may edit any entry of the query model in order to optimize retrieval results by iteration.