We propose a ray-based segmentation method for color images. A segment is represented by a centroid and evenlydistributed rays shooting out from it. First, a bottom-up low-level boundary detection process coarsely constructs candidate segments. Then, two top-down learning processes, mid-level intra-segment learning and high-level inter-segment learning, create the best segments. Segments are created sequentially until all pixels are classified. The number of segments is determined automatically. We test our method on the Berkeley Segmentation Dataset. Evaluation results show that our algorithm produces better results than those of the Normalized Cuts segmentation method.