The augmented desk is gaining popularity in recent HCI research. Its layout of a large horizontal screen on the desk enhances immersive and intense collaborative experiences. A responsive and unimpeded input interface is important for an efficient interaction in such an environment. In this paper, we developed a real-time stereo vision-based finger pointing interface for our augmented desk that supports drag-and-drop operation by the tap-and-move of the index finger. The core of our system is a 3D fingertip tracking system, which requires both careful calibration and efficient fingertip localization algorithm. To meet these requirements, we have designed a two-step calibration method that strikes a good balance between accuracy and convenience. Furthermore, based on the chain code representation of the contour, we propose the direction cancellation vector as a tool for fingertip localization. Our algorithm works efficiently with a time complexity of )(nO in term of the length of the c...