We update the SVM score of an object through a video sequence with a small and variable subset of support vectors. In the first frame we use all the support vectors to compute the SVM score of the object but in subsequent frames we use only a small and variable subset of support vectors to update the SVM score. In each frame we calculate the dot-products of the support vectors in the subset with the pattern of the object being tracked. The difference in the dot-products, between past and current frames, is used to update the SVM score. This is done at a fraction of the computational cost required to re-evaluate the SVM score from scratch in every frame. The two methods we develop are "Cyclic subset selection", in which we break the set of all support vectors into subsets of equal size and use them cyclically, and "Maximum variance subset selection", in which we choose the support vectors whose dot-product with the test pattern varied the most in previous frames. We...