In this paper, we present a physically-based simulation framework to build up real-time Minimally invasive surgery (MIS) simulators using pointbased techniques. In our framework, simulations of organ contacts, collisions, deformations and haptic feedbacks are all based on discrete physics points. We propose a new smoothed particle hydrodynamics (SPH) approach to simulate non-linear biological soft tissues with a specified tensor computation scheme and experimentally measured organ parameters. We employ a uniform grid method to handle collisions and contacts between organs and surgical instruments represented by particles in real-time. Additionally, a point-based smoothing method for contact feedback is proposed. Our framework could simulate complex surgical scenes in MIS simulation in a unified way, and improve the time efficiency of the entire system without loss of physics accuracy, as shown in our experiment results.