Finding correspondences of two images taken from largely different camera configuration is a challenging problem because appearance information such as color, intensity and edge orientation histogram cannot be used. A probabilistic approach to match line segments in the images is proposed for calculating their homography. A membership matrix to represent contribution of every match to the homography is employed. A relaxed version of forward stepwise regression is derived and the test statistic is optimized. An alternating scheme for optimizing the membership and homography is provided. The simulation results on synthetic images validate the proposed method.