This paper proposes a robust statistical framework to extract highlights from a baseball broadcast video. We applied multistream Hidden Markov Models (HMMs) to control the weights among different features. To achieve robustness against new highlights, we used a common simple structure for all the HMMs. In addition, scene segmentation and unsupervised adaptation were applied to achieve more robustness against the differences of environmental conditions among games. The precision rate of highlight extracting experiments for eight kinds of highlights from 4.5 hours of digest data was 77.4% and was increased to 78.7% by applying scene segmentation. Futhermore, the unsupervised adap