In this paper, we present a technique for tracking video objects in a sequence. The proposed technique is based on a backward projection technique. Since classical backward technique can be disturbed by occlusions and potential errors in spatial segmentation or motion estimation, we propose an extension to the backward projection technique in order to cope with these problems. Results obtained show the relevance of the proposed approach for various kind of tracked objects that can either be rigid or non-rigid.