The nature of the concepts regarding images in many domains are imprecise, and the interpretation of finding similar images is also ambiguous and subjective on the level of human perception. To solve these problems, in this paper, images’ semantic categories and the tolerance degree between them are defined systematically, and the approach of modeling tolerance relations between the semantic classes is proposed. Furthermore, for removing the induced false tolerance in the produce of using semantic tolerance relation model, the method of un-tolerating is introduced in image representation. We apply the proposed approach to the representations of images regarding the nature vs. man-made domain, human vs. nonhuman domain, and temporal domain, and compare the categorization results of them with the results not using semantic tolerance relation model. The results show the effectiveness of proposed method.