Variable Word Rate N-grams

The rate of occurrence of words is not uniform but varies from document to document. Despite this observation, parameters for conventional n-gram language models are usually derived using the assumption of a constant word rate. In this paper we investigate the use of variable word rate assumption, modelled by a Poisson distribution or a continuous mixture of Poissons. We present an approach to estimating the relative frequencies of words or ngrams taking prior information of their occurrences into account. Discounting and smoothing schemes are also considered. Using the Broadcast News task, the approach demonstrates a reduction of perplexity up to 10%.

• Type: Journal
• Year: 2000
• Where: CORR
• Authors: Yoshihiko Gotoh, Steve Renals
• Keywords: Constant Word Rate, CORR 2000, Education, Word Rate, Word Rate Assumption