Human recognition from video requires solving the two tasks, recognition and tracking, simultaneously. This leads to a parameterized time series state space model, representing both motion and identity of the human. Sequential Monte Carlo (SMC) algorithms, like CONDENSATION [3], can be developed to offer numerical solutions to this model. However, in outdoor environments, the solution is more likely to diverge from the foreground, causing failures in both recognition and tracking. In this paper, we propose an approach for tackling this problem by incorporating the constraint of temporal continuity in the observations. Experimental results demonstrate improvements over its CONDENSATION counterpart.