Establishing point correspondences between images is a key step for 3D-shape computation. Nevertheless, shape extraction and point correspondence are treated, usually, as two different computational processes. We propose a new method for solving the correspondence problem between points of a fully uncalibrated scaled-orthographic image sequence. Among all possible point selections and permutations, our method chooses the one that minimizes the fourth singular value of the observation matrix in the factorization method. This way, correspondences are set such that shape and motion computation are optimal. Furthermore, we show this is an optimal criterion under bounded noise conditions. Also, our formulation takes feature selection and outlier rejection into account, in a compact and integrated way. The resulting combinatorial problem is cast as a concave minimization problem that can be efficiently solved. Experiments show the practical validity of the assumptions and the overall perform...