Object tracking is one of the fundamental problems in computer vision and has received considerable attention in the past two decades. The success of a tracking algorithm relies on two key issues: 1) an effective representation so that the object being tracked can be distinguished from the background and other objects and 2) an update scheme of the object representation to accommodate object appearance and structure changes. Despite the progress made in the past, reliable and efficient tracking of objects with changing appearance remains a challenging problem. In this paper, a novel sparse, local feature-based object representation, the attributed relational feature graph, is proposed to solve this problem. The object is modeled using invariant features such as the scale-invariant feature transform and the geometric relations among features are encoded in the form of a graph. A dynamic model is developed to evolve the feature graph according to the appearance and structure changes by a...