Video shots provide the most basic meaningful segments for video analysis and understanding. In this paper, we present a detection and classification framework for the video shot segmentation in a coarse-to-fine fashion. The initial transitions are detected from a sub-sampled video space. These coarse segments are later refined in the original video space with the technique of illumination artifacts removal and transition finalization. The transition type (abrupt or gradual) are finally determined by exploiting the histogram intersection plot. The proposed method has been tested on a large amount of videos, which contain a variety of types of shot transitions. Accurate and competitive results have been obtained.