We present an algorithm that quickly and accurately estimates vanishing points in images of man-made environments. Contrary to previously proposed solutions, ours is neither iterative nor relies on voting in the space of vanishing points. Our formulation is based on a recently proposed algorithm for the simultaneous estimation of multiple models called J-Linkage. Our method avoids representing edges on the Gaussian sphere and the computations and error measures are done in the image. We show that a consistency measure between a vanishing point and an edge of the image can be computed in closed-form while being geometrically meaningful. Finally, given a set of estimated vanishing points, we show how this consistency measure can be used to identify the three vanishing points corresponding to the Manhattan directions. We compare our algorithm with other approaches on the York Urban Database and show significant performance improvements.