Content-based retrieval (CBIR) methods in medical databases have been designed to support specific tasks, such as retrieval of digital mammograms or 3D MRI images. These methods cannot be transferred to other medical applications since different imaging modalities require different types of processing. To enable content-based queries in diverse collections of medical images, the retrieval system must be familiar with the current image class prior to the query processing. We describe a novel approach for the automatic categorization of medical images according to their modalities. We propose a semantically based set of visual features, their relevance and organization for capturing the semantics of different imaging modalities. The features are used in conjunction with a new categorization metric, enabling "intelligent" annotation, browsing/searching of medical databases. Our algorithm provides basic semantic knowledge about the image, and may serve as a front-end to the doma...