This paper demonstrates how a simple, yet effective, set of features enables to integrate ensemble classifiers in optical flow based tracking. In particular, gray value differences of pixel pairs are used for generating binary weak classifiers, forming the respective object representation. For the tracking step an affine motion model is proposed. By using hinge loss functions, the motion estimation problem can be formulated as a linear program. Experiments demonstrate robustness of the proposed approach and include comparisons to conventional tracking methods.