In this paper a prototype system is described for the management and content-based retrieval of defect images in huge image databases. This is a real problem in surface inspection applications, since modern inspection systems may produce up to thousands of defect images in a day. The retrieval is based on shape and internal structure characteristics of defects, so no manual labeling nor annotation needs to be made. We are using a noncommercial, generic content-based image retrieval (CBIR) system called PicSOM that is modified to fit to the special requirements of our application. The results of experiments show that the prototype system works fast with good retrieval results.