— suppose we have a video, the first half of this video is capturing the images of a sedan and the second half is recording the moving of a truck, can we use the same video tracking algorithm to follow the moving of the object over the entire video sequence? We define this kind of problem as “Object Class Tracking” problem. Instead of tracking a specific object, object class tracking is to track the moving of the object class. The challenge is how to locate the image element in the next frame by handling the large intra-class variance. In this paper, we propose a theoretic framework for object class tracking based on Kalman Filter. A part-based statistical model is employed to solve the image element localization problem. We mathematically prove the soundness of the theoretic framework. The method has the potential to be applied in many application domains.