—In this work, we make use of 3D visual contours carrying geometric as well as appearance information. Between these contours, we define 3D relations that encode structural information relevant to object-level operations such as similarity assessment and grasping. We show that this relational space can also be used as input features for learning which we exemplify for the grasping of unknown objects. Our representation is motivated by the human visual system in two respects. First, we make use of a visual descriptor that is motivated by hyper-columns in