—We present a method to determine 3D motion and structure of multiple objects from two perspective views, using adaptive Hough transform. In our method, segmentation is determined based on a 3D rigidity constraint. Instead of searching candidate solutions over the entire five-dimensional translation and rotation parameter space, we only examine the two-dimensional translation space. We divide the input image into overlapping patches, and, for each sample of the translation space, we compute the rotation parameters of patches using least-squares fit. Every patch votes for a sample in the fivedimensional parameter space. For a patch containing multiple motions, we use a redescending M-estimator to compute rotation parameters of a dominant motion within the patch. To reduce computational and storage burdens of standard multidimensional Hough transform, we use adaptive Hough transform to iteratively refine the relevant parameter space in a “coarse-to-fine” fashion. Our method can rob...