In this paper we present a probabilistic framework for tracking objects based on local dynamic segmentation. We view the segn to be a Markov labeling process and abstract it as a MAP problem. In the Bayesian formulation, we exploit the Feature-Spatial-Measure distribution of local area as the conditional distribution. The Feature-Spatial vector is used to constrain the appearance of region while the Measure vector is used to constrain the label of the pixels in the region. One drive force to the introduction of FSM distribution is the HMMF model that makes it possible to estimate the Measure field by the minimization of a differentiable function. Mean-shift procedure and IFGT technique are used to further alleviate the computational costs. Very promising experimental results on synthetic and natural sequences are presented to illustrate the performance of the presented algorithm.