We present a method for the construction of patient-specific atlases of the brain. Traditional atlases of the brain aim to characterize the variability of a population of subjects. A common approach is to average the anatomies of a population after alignment to a common coordinate system. Subjects are typically given equal weights during averaging which results in atlases that are population-specific rather than subject specific. In this paper we propose a method for the construction of patient-specific atlas for a given query subject from a large population cohort. During the atlas construction we compute the similarity between the query subject and the subjects in the population cohort. This similarity measure can be based on image similarity or other meta-information (e.g. sex, age, ethnicity, medical history, etc). We show an example of the construction of brain atlases for different ages using a cohort of 575 subjects between the ages of 18 and 80.