Many computer vision problems rely on computing histogram-based objective functions with a sliding window. A main limiting factor is the high computational cost. Existing computational methods have a complexity linear in the histogram dimension. In this paper, we propose an efficient method that has a constant complexity in the histogram dimension and therefore scales well with high dimensional histograms. This is achieved by harnessing the spatial coherence of natural images and computing the objective function in an incremental manner. We demonstrate the significant performance enhancement by our method through important vision tasks including object detection, object tracking and image saliency analysis. Compared with stateof-the-art techniques, our method typically achieves from tens to hundreds of times speedup for those tasks.