In urban areas, buildings are often used as landmarks for localization. Reliable and efficient recognition of buildings is crucial for enabling this functionality. Motivated by the applications which would enhance visual localization and navigation capabilities we propose in this paper a hierarchical approach for building recognition. In the first recognition stage the model views are indexed by localized color histograms computed from dominant orientation structures in the image. This novel representation enables quick retrieval of a small number of candidate buildings from the database. In the second stage the recognition results are refined by matching previously proposed SIFT descriptors associated with local image regions. For this stage we propose a method for selecting discriminative SIFT features and a simple probabilistic model for integration of the evidence from individual matches based on the match quality. This enables us to eliminate the sensitive choice of threshold ...