In medical imaging, constructing an atlas and bringing an image set in a single common reference frame may easily lead the analysis to erroneous conclusions, especially when the population under study is heterogeneous. In this paper, we propose a framework based on spectral clustering that is capable of partitioning an image population into sets that require a separate atlas, and identifying the most suitable templates to be used as coordinate reference frames. The spectral analysis step relies on pairwise distances that express anatomical differences between subjects as a function of the diffeomorphic warp required to match the one subject onto the other, plus residual information. The methodology is validated numerically on artificial and medical imaging data.