A model-driven method for handling clothes by two manipulators based on observation with stereo cameras is proposed. The task considered in this paper is to hold up a specific part of clothes (e.g. one shoulder of a pullover) by the second manipulator, when the clothes is held in the air by the first manipulator. First, the method calculates possible 3D shapes of the hanging clothes by simulating the clothes deformation. The 3D shape whose appearance gives the best fit with the observed appearance is selected as estimation of the current state. Then, based on the estimated shape, the 3D position and normal direction of the part where the second manipulator should hold are calculated. The experiments using actual two manipulators have shown the good potential of the proposed method.