Physical limits of inference