A split-and-merge framework based on a maximum variance criterion is proposed for disparity clustering. The proposed algorithm transforms low-level stereo disparity information to mid-level planar surface information which can be used further to carry out high-level computer vision tasks such as shape classification. Unlike conventional clustering, the proposed algorithm assumes that the number of clusters is unknown. Instead, a maximum variance criterion is applied to extract planar surfaces from the disparity image. The split phase of the algorithm creates clusters based on spatial continuity and the merge phase combines these clusters such that variance per cluster does not exceeded an allowable value. For efficient maximum variance clustering, a greedy branch-and-bound procedure is introduced. Efficiency of the approach is verified through experiments.