Abstract Kernel-based object tracking refers to computing the translation of an isotropic object kernel from one video frame to the next. The kernel is commonly chosen as a primitive geometric shape and its translation is computed by maximizing the likelihood between the current and past object observations. In the case when the object does not have an isotropic shape, kernel includes non-object regions which biases the motion estimation and results in loss of the tracked object. In this paper, we propose to use an asymmetric object kernel for improving the tracking performance. An important advantage of an asymmetric kernel over an isotropic kernel is its precise representation of the object shape. This property enhances tracking performance due to discarding the non-object regions. The second contribution of our