Knowing the Unseen: Estimating Vocabulary Size over Unseen Samples

Empirical studies on corpora involve making measurements of several quantities for the purpose of comparing corpora, creating language models or to make generalizations about specific linguistic phenomena in a language. Quantities such as average word length are stable across sample sizes and hence can be reliably estimated from large enough samples. However, quantities such as vocabulary size change with sample size. Thus measurements based on a given sample will need to be extrapolated to obtain their estimates over larger unseen samples. In this work, we propose a novel nonparametric estimator of vocabulary size. Our main result is to show the statistical consistency of the estimator

• Type: Journal
• Year: 2009
• Where: ACL
• Authors: Suma Bhat, Richard Sproat
• Keywords: ACL 2009, Computational Linguistics, Estimator, Sample Size, Vocabulary Size