Reconstructing 3D scenes with independently moving objects from uncalibrated monocular image sequences still poses serious challenges. One important problem is to find the relative scales between these different reconstructed objects. The perspective reconstruction of a single object can only be known up to a certain scale which results in a one-parameter family of relative scales and trajectories per moving object. The paper formulates this ambiguity and proposes solutions in the vein of "non-accidentalness". Two instantiations of its use are analyzed: planar motion and the 'heading constraint'.