Abstract In this paper, we present a framework for the classification of images in surface inspection tasks and address several key aspects of the processing chain from the original image to the final classification result. A major contribution of this paper is a quantitative assessment of how incorporating adaptivity into the feature calculation, the feature pre-processing,andintotheclassifiersthemselves,influences the final image classification performance. Hereby, results achieved on a range of artificial and real-world test data from applications in printing, die-casting, metal processing and food production are presented.