This paper presents an approach to collision detection and response for dynamically deforming pointbased objects. Both the volume of an object and its surface are represented by point sets. In case of a collision, response forces are computed for penetrating surface points and distributed to volume points which are used for simulating the object dynamics. The decoupling of collision handling and deformation allows for a very stable collision response while maintaining interactive update rates of the dynamic simulation for environments with moderate complexity. Simulation results are presented for elastically and plastically deforming objects with changing topology.