Particle filter is a sequential Monte Carlo method for object tracking in a recursive Bayesian filtering framework. The efficiency and accuracy of the particle filter depends on two key factors: how many particles are used and how these particles are re-located. In this paper, we estimate the number of required particles using the Kullback-Leibler distance (KLD), which is called KLD-sampling, and we use a hybrid dynamic model to generate diversified particles, which suits object’s agile motion. Besides, we employ the mean shift analysis as a local mode seeking mechanism to make each particle more “informative”. We demonstrate the performance of the proposed algorithm tracking the ball in sports video clips.