This paper presents a novel semantic-oriented video analysis system for the basketball game videos. Based on Bayesian Belief Network (BBN), it may bridge this gap between the low-level features describing image/video structure and the high-level knowledge. We apply the Support Vector Machine (SVM) to identify and track the ball, the shooter, and the basket as the lowlevel features. Based on these features, our BBN framework can identify four categories of shot event such as short shot, medium shot, long shot, free throw, and the scoring event. In the experiments, we demonstrate that our system may interpret the video shots in terms of four different shot events and one scoring event effectively.