— In this paper, a new collaborative tracking algorithm is put forward to track multiple objects in video streams. First, we suggest a robust color-based tracker whose model is updated by online learned contextual information. A recursive method is performed to improve the estimation accuracy and the robustness to cluttered environment. Then, we extend this tracker to multiple targets. In order to avoid the problem of ID switch in long term occlusion, we design a hierarchical tracking system with different tracking priorities. First, the algorithm employs an adaptive collision prevention model to separate the nearby trajectories. When the inter-occlusion happens, the holistic model of tracker splits into several parts, and we use the visible parts to perform tracking as well as occlusion reasoning. In case where the targets have close appearance models, a trajectory monitoring approach is employed to handle the occlusion. Once the tracker is fully occluded, the algorithm will re-init...