In this paper, we propose a novel approach which integrates adaptive appearance model and hierarchical estimation mechanism composed of global estimation and local estimation. Hierarchical estimation runs in two phases: In first phase, global estimation coarsely predicts a region in where true state may be present, and then local estimation tries to find out the true state inside the region at second phase. The benefits from Hierarchical estimation are two-fold, on one hand, it reduces the number of particles significantly, which enables real-time tracking, while on the other hand, it improves tracking accuracy even with less number of particles. Experimental results show the effectiveness and robustness of the proposed approach.