Color is very useful in locating and recognizing objects that occur in artificial environments. The color histogram has shown its efficiency and advantages as a general tool for various applications, such as content-based image retrieval and video browsing, object indexing and location, and video segmentation. However, due to the lack of any spatial and context information, the histogram is not robust and effective for color characterization (e.g. dominant color) in large video databases. In this paper, we propose a nonparametric color characterization model using mean shift procedure, with an emphasis on spatio-temporal consistency. Experimental results suggest that the color characterization model is much more effective for video indexing and browsing, particularly in the domain of structured video (e.g. sports video). Categories and Subject Descriptors I.4.7 [Image Processing and Computer Vision]: Scene Analysis – Color. General Terms: Algorithms.