The paper describes a simple but effective framework for visual object tracking in video sequences. The main contribution of this work lies in the introduction of a case-based reasoning (CBR) method to maintain an accurate target model automatically and efficiently under significant appearance changes without drifting away. Specifically, an automatic case-base maintenance algorithm is proposed to dynamically update the case base, manage the case base to be competent and representative, and to maintain the case base in a reasonable size for real-time performance. Furthermore, the method can provide an accurate confidence measurement for each tracked object so that the tracking failures can be identified in time. Under the framework, a real-time face tracker is built to track human faces robustly under various face orientations, significant facial expressions, and illumination changes.