Although more efficient in computation compared to other tracking approaches such as particle filtering, the kernel-based tracking suffers from the "singularity" problem which makes the tracking unstable and even completely fail. In this paper, we propose a novel framework to handle this problem by enhancing the tracker's observability. In particular, we formulate object tracking as an inverse problem, thus unifying the existing kernel-based tracking approaches into a consistent theoretical framework. By exploiting the observability theory, we explicitly give the criterion for kernel design and constraint selection. Moreover, we extend the kernel-based approach by including the state dynamics and thus form a state-space model. The use of observability theory is also extended for dynamics estimation and evaluation. We rely on an optimal observer for state estimation as a solution to video tracking. The performance of the proposed approach has been demonstrated on both sy...