Always Good Turing: Asymptotically Optimal Probability Estimation

While deciphering the Enigma Code during World War II, I.J. Good and A.M. Turing considered the problem of estimating a probability distribution from a sample of data. They derived a surprising and unintuitive formula that has since been used in a variety of applications and studied by a number of researchers. Borrowing an information-theoretic and machine-learning framework, we define the attenuation of a probability estimator as the largest possible ratio between the per-symbol probability assigned to an arbitrarily-long sequence by any distribution, and the corresponding probability assigned by the estimator. We show that some common estimators have infinite attenuation and that the attenuation of the GoodTuring estimator is low, yet larger than one. We then derive an estimator whose attenuation is one, namely, as the length of any sequence increases, the per-symbol probability assigned by the estimator is as high as possible. Interestingly, some of the proofs use celebrated resu...

• Type: Conference
• Year: 2003
• Where: FOCS
• Authors: Alon Orlitsky, Narayana P. Santhanam, Junan Zhang
• Keywords: Attenuation, Estimator, FOCS 2003, Per-symbol Probability, Theoretical Computer Science