In this paper we present a new scheme for detection and tracking of specific objects in a knowledge-based framework. The scheme uses a supervised learning method: Support Vector Machines. Both problems, detection and tracking, are solved by a common approach: objects are located in video sequences by a SVM classifier. They are next tracked along the time by a SVM tracker with complete 6 parameters affine model. The method is applied in a video surveillance application for detection and tracking of frontal view faces. Real time application constraints are met by reduction of support vector set.