In this paper, we propose a physically-based dynamical model for tracking. Our model relies on Newton’s second law of motion, which governs any real-world dynamical system. As a consequence, it can be generally applied to very different tracking problems. Furthermore, since the equations describing Newton’s second law are simple linear equalities, they can be incorporated in any tracking framework at very little cost. Leveraging this lets us introduce a convex formulation of 3D tracking from monocular images. We demonstrate the strengths of our approach on various types of motion, such as billiards and acrobatics.