This paper develops a weakly supervised algorithm that learns to segment rigid multi-colored objects from a set of training images and key points. The approach uses congealing to learn a probabilistic spatial model of the multi-colored object class and graph-cut to separate the foreground from the background. The result is a novel approach which can segment heterogeneous objects, in contrast to other recent approaches which are better at segmenting uniform but possibly flexible objects.