Traditional dynamical systems used for motion tracking cannot effectively handle high dimensionality of the motion states and composite dynamics. In this paper, to address both issues simultaneously, we propose the marriage of the switching dynamical system and recent Gaussian Process Dynamic Models (GPDM), yielding a new model called the switching GPDM (SGPDM). The proposed switching variables enable the SGPDM to capture diverse motion dynamics effectively, and also allow to identify the motion class (e.g. walk or run in the human motion tracking, smile or angry in the facial motion tracking), which naturally leads to the idea of simultaneous motion tracking and classification. Moreover, each of GPDMs in SGPDM can faithfully model its corresponding primitive motion, while performing tracking in the low-dimensional latent space, therefore significantly improving the tracking efficiency. The proposed SGPDM is then applied to human body motion tracking and classification, and facial mot...