Enormous accessible broadcast soccer videos demand an efficient ball and player trajectory extraction framework to represent the tactic semantics for the automatic analysis. Camera motions, noise and blurs in broadcast videos make it difficult to extract the trajectories with a single existing object tracking algorithm. In this paper, we propose a novel framework for ball and player trajectory extraction in broadcast soccer videos. The framework generates candidate ball trajectories and player trajectory segments, then it searches the optimal ball trajectory with the likelihood ranking and refines player trajectories with MCMC data association. Instead of extracting ball and player trajectories respectively, our framework employs the motion relationship of the ball and players to build a collaborate scheme to improve the tracking and trajectory refinement results. The experimental results show the proposed framework is more effective than previous works.