This paper presents an overview of our project on an augmented desk interface system called the EnhancedDesk. The EnhancedDesk is equipped with an infrared camera for tracking a users' hand motions, a color video camera for recognizing objects, and projectors for displaying various kinds of digital information onto the desk. The key technical innovations of the EnhancedDesk include: fast and accurate tracking of multiple fingers for direct manipulation of both real and projected objects with bare hands, interactive object registration and recognition with hand gestures, vision-based tracking of a user's gaze, and addition of interactive functionality overlaid on the desk. This paper presents a brief description of the components, as well as some prototypes of applications of our augmented desk interface system.