Low-level image processing algorithms generally provide noisy features that are far from being Gaussian. Medium-level tasks such as object detection must therefore be robust to outliers. This can be achieved by means of the well-known M-estimators. However, higher-level systems do not only need robust detection, but also a confidence value associated to the detection. When the detection is cast into the fitting framework, the inverse of the covariance matrix of the fit provides a valuable confidence matrix. Since there is no closed-form expression of the covariance matrix in the robust case, one must resort to some approximation. Unfortunately, the experimental evaluation reported in this paper on real data shows that, among the different approximations proposed in literature that can be efficiently computed, none provides reliable results. This leads us to study the robustness of the covariance matrix of the fit with respect to noise model parameters. We introduce a new non-asymptotic...