One of the major challenges in stereo matching is to handle partial occlusions. In this paper, we introduce the Outlier Confidence (OC) which dynamically measures how likely one pixel is occluded. Then the occlusion information is softly incorporated into our model. A global optimization is applied to robustly estimating the disparities for both the occluded and non-occluded pixels. Compared to color segmentation with plane fitting which globally partitions the image, our OC model locally infers the possible disparity values for the outlier pixels using a reliable color sample refinement scheme. Experiments on the Middlebury dataset show that the proposed two-frame stereo matching method performs satisfactorily on the stereo images.