Recent advances in surgical robotics have provided a platform for extending the current capabilities of minimally invasive surgery by incorporating both pre-operative and intra-operative imaging data. In this tutorial paper, we introduce techniques for in vivo 3D tissue deformation recovery and tracking, based on laparoscopic or endoscopic images. These optically based techniques provide a unique opportunity for recovering surface deformation of the soft tissue without the need of additional instrumentation. They can therefore be easily incorporated into the existing surgical workflow. Technically, the problem formulation is challenging due to non-rigid deformation of the tissue and instrument interaction. Current approaches and future research directions in terms of intra-operative planning and adaptive surgical navigation are explained in detail.