Color-based tracking methods have proved to be efficient for their robustness qualities. The drawback of such global representation of an object is the lack of information on its spatial configuration, making difficult the tracking of more complex motions. This issue can be overcome by using several kernels weighting pixels locations. In this paper a multiple kernels configuration is proposed and developed in both probabilistic and deterministic frameworks. The advantages of both approaches are combined to design a robust tracker allowing to track location, size and orientation of the object. A target tracking scheme using visual servoing considering measurements provided by the presented approach validates the proposed method.