Image segmentation and its performance evaluation are very difficult but important problems in computer vision. A major challenge in segmentation evaluation comes from the fundamental conflict between generality and objectivity: For general-purpose segmentation, the ground truth and segmentation accuracy may not be well defined, while embedding the evaluation in a specific application, the evaluation results may not be extended to other applications. We present in this paper a new benchmark for evaluating image segmentation. Specifically, we formulate image segmentation as identifying the single most perceptually salient structure from an image. We collect a large variety of test images that conforms to this specific formulation, construct unambiguous ground truth for each image, and define a reliable way to measure the segmentation accuracy. We then present two special strategies to further address two important issues: (a) the most salient structures in some real images may not be u...