In this paper, a video segmentation algorithm based on Hidden Markov Model classifier with multimodal feature is proposed. By using Hidden Markov Model classifier with both audio and visual features, erroneous shot boundary detection and over-segmentation were avoided compared with conventional algorithms. The experimental results show that the propose method is effective in shot detection.