Despite a lot of research efforts in sports video processing, little work was done in volleyball video analysis due to the high density of players on the court and the complicated overlapping of player-player or ball-player, which lead to great complexity of object tracking for advanced analysis. For ball detection and trajectory extraction in volleyball videos, this paper presents a physics-based scheme which utilizes the motion characteristics to extract ball trajectory from lots of moving objects. The experiments of ball tracking show encouraging results. Moreover, based on gamespecific properties, the ball trajectory can be exploited to recognize set types for tactics inference and to detect basic actions in the volleyball game for close-up presentation.