We introduce a stochastic model to characterize the online computational process of an object recognition system based on a hierarchy of classifiers. The model is a graphical network for the conditional distribution, under both object and background hypotheses, of the classifiers which are executed during a coarse-to-fine search. A likelihood is then assigned to each history or "trace" of processing. In this way, likelihood ratios provide a measure of confidence for each candidate detection, which markedly improves the selectivity of hierarchical search, as illustrated by pruning many false positives in a face detection experiment. This also leads to a united framework for object detection and tracking. Experiments in tracking faces in image sequences demonstrate invariance to large face movements, partial occlusions, changes in illumination and varying numbers of faces.