In this paper, we first review our work in the domain of dextrous manipulation, where we introduced Manipulation Manifolds – a highly structured manifold representation of hand postures which lends itself to simple and robust manipulation control schemes. Coming from this scenario, we then present our idea of how this generative system can be naturally extended to the recognition and segmentation of the represented movements providing the core representation for a combined system for action production and recognition.