Estimating planar projective transform (homography) from a pair of images is a classical problem in computer vision. In this paper, we propose a novel algorithm for direct registering two point sets in R2 using projective transform without using intensity values. In this very general context, there is no easily established correspondences that can be used to estimate the projective transform, and most of the existing techniques become either inadequate or inappropriate. While the planar projective transforms form an eight-dimensional Lie group, we show that for registering 2D point sets, the search space for the homographies can be effectively reduced to a three-dimensional space. To further improve on the running time without significantly reducing the accuracy of the registration, we propose a matching cost function constructed using local polynomial moments of the point sets and a coarse to fine approach. The resulting registration algorithm has linear time complexity with respect t...