This paper proposes a data driven image segmentation algorithm, based on decomposing the target output (ground truth). Classical pixel labeling methods utilize machine learning algorithms that induce a mapping from pixel features to individual pixel labels. In contrast we propose to first extract features from both images and labels. Subsequently we induce a mapping from pixel features to label features and synthesize the final output by combining the newly derived label components. We demonstrate the effectiveness of the proposed approach by applying log-Gabor filters to both input and ground truth images of mineral ore. Subsequently we train perceptrons and regression trees to produce individual output components that are combined in frequency space to create the final segmentation. Experimental results show significant improvements over contextual pixel labeling and over ensemble methods.