Medical CBIR (content-based image retrieval) applications pose unique challenges but at the same time offer many new opportunities. On one hand, while one can easily understand news or sports videos, a medical image is often completely incomprehensible to untrained eyes. On the other hand, semantics in the medical domain is much better defined and there is a vast accumulation of formal knowledge representations that could be exploited to support semantic search for any specialty areas in medicine. In this paper, however, we will not dwell on any one particular specialty area, but rather address the question of how to support scalable semantic search across the whole of medical CBIR field: what are the advantages to take and gaps to fill, what are the key enabling technologies, and the critical success factor from an industrial point of view.