Instance-based learning (IBL) algorithms have proved to be successful in many applications. However, as opposed to standard statistical methods, a prediction in IBL is usually given without characterizing its confidence. In this paper, we propose an IBL method that allows for deriving set-valued predictions that cover the correct answer (label) with high probability. Our method makes use of a formal model of the heuristic inference principle suggesting that similar instances do have similar labels. The focus of this paper is on the prediction of numeric values (regression), even though the method is also useful for classification problems if a reasonable similarity measure can be defined on the set of classes.