— Classical position-based visual servoing approaches rely on the presence of distinctive features in the image such as corners and edges. In this contribution we exploit a hierarchical approach for object detection, initial-pose estimation, and realtime tracking based first on colour distribution and subsequently on the shape and texture information. The shape model of the object is not limited to surface primitives but allow for any free-form surface not subject to self-occlusion. We evaluate the approach as part of a handshake scenario where a 7-DoF robot takes a free moving object over from a human.