In this paper we develop a theory of non-parametric self-calibration. Recently, schemes have been devised for non-parametric laboratory calibration, but not for selfcalibration. We allow an arbitrary warp to model the intrinsic mapping, with the only restriction that the camera is central and that the intrinsic mapping has a well-defined non-singular matrix derivative at a finite number of points under study. We give a number of theoretical results, both for infinitesimal motion and finite motion, for a finite number of observations and when observing motion over a dense image, for rotation and translation. Our main result is that through observing the flow induced by three instantaneous rotations at a finite number of points of the distorted image, we can perform projective reconstruction of those image points on the undistorted image. We present some results with synthetic and real data.