Self-calibrating Probability Forecasting

In the problem of probability forecasting the learner’s goal is to output, given a training set and a new object, a suitable probability measure on the possible values of the new object’s label. An on-line algorithm for probability forecasting is said to be well-calibrated if the probabilities it outputs agree with the observed frequencies. We give a natural nonasymptotic formalization of the notion of well-calibratedness, which we then study under the assumption of randomness (the object/label pairs are independent and identically distributed). It turns out that, although no probability forecasting algorithm is automatically well-calibrated in our sense, there exists a wide class of algorithms for “multiprobability forecasting” (such algorithms are allowed to output a set, ideally very narrow, of probability measures) which satisfy this property; we call the algorithms in this class “Venn probability machines”. Our experimental results demonstrate that a 1-Nearest Neighbo...

• Type: Conference
• Year: 2003
• Where: NIPS
• Authors: Vladimir Vovk, Glenn Shafer, Ilia Nouretdinov
• Keywords: NIPS 2003, NIPS 2007, Probability Forecasting, Probability Measure, Venn Probability Machine