Most image retrieval systems perform a linear search over the database to find the closest match to a query. However, databases usually exhibit a natural grouping structure into content classes that can be exploited to improve retrieval precision and speed. We investigate methods that enable search at both the class and image level. It is shown that, through the combination of Bayesian averaging and hierarchical density estimation, it is possible to achieve significant gains in retrieval accuracy and speed, at the cost of a marginal increase in training complexity. The technique is also shown to enable the efficient design of semantic classifiers.