We propose a novel approach to designing algorithms for
object tracking based on fusing multiple observation models.
As the space of possible observation models is too large
for exhaustive on-line search, this work aims to select models
that are suitable for a particular tracking task at hand.
During an off-line training stage observation models from
various off-the-shelf trackers are evaluated. From this data
different methods of fusing the observers on-line are investigated,
including parallel and cascaded evaluation. Experiments
on test sequences show that this evaluation is useful
for automatically designing and assessing algorithms for a
particular tracking task. Results are shown for face tracking
with a handheld camera and hand tracking for gesture
interaction. We show that for these cases combining a small
number of observers in a sequential cascade results in efficient
algorithms that are both robust and precise.