Video classi cation and segmentation are fundamental steps for e cient accessing, retrieving and browsing large amount of video data. We have developed a scene classi cationscheme using a Hidden MarkovModel HMMbased classi er. By utilizing the temporal behaviors of di erent scene classes, HMM classi er can e ectively classify video segments into one of the prede ned scene classes. In this paper, we describe two approaches for joint video classi cation and segmentation based on HMM, which works by searching for the most likely class transition path utilizing the dynamic programming technique.