Cardiopulmonary imaging is a key tool in modern diagnostic and interventional medicine. Automated analysis of MRI or ultrasound video is complicated by limitations on the image quality and complicated deformations of the chest cavity created by patient breathing and heart beating. When these are the primary causes of image variation, the video sequence samples a two-dimensional, nonlinear manifold of images. Nonparametric representations of this image manifold can be created using recently developed manifold learning algorithms. For automated analysis tasks that require segmenting many images, this manifold structure provides strong new cues on the shape and deformation of particular regions of interest. This paper develops the theory and algorithms to incorporate these manifold constraints within a level set based segmentation algorithm. We apply our algorithm, based on manifold constraints to the problem of segmenting the left ventricle, and show the improvement that arises from usin...