In previous work we have presented a prototype of an assistant system for the blind that can be used for self-localization and interactive object identification of static objects stored within 3D environment models. In this paper we present a new method for interactive tracking of various types of movable objects. The state of fixed movable objects, like doors, can be recognized by comparing the distance between sensor data and a 3D model. For the identification and model-based tracking of free movable objects, like chairs, we have developed an algorithm that is similar to human perception, based on shape and color comparisons to trained objects. Further, using a common face detection algorithm, our assistant system informs the user of the presence of people, and enables the localization of a real person based on interactive tracking of virtual models of humans. Categories and Subject Descriptors H.5.2 [User Interfaces]: User-centered design, Prototyping I.4.6 [Segmentation]: Edge and...