Randomized Language Models via Perfect Hash Functions

We propose a succinct randomized language model which employs a perfect hash function to encode fingerprints of n-grams and their associated probabilities, backoff weights, or other parameters. The scheme can represent any standard n-gram model and is easily combined with existing model reduction techniques such as entropy-pruning. We demonstrate the space-savings of the scheme via machine translation experiments within a distributed language modeling framework.

• Type: Conference
• Year: 2008
• Where: ACL
• Authors: David Talbot, Thorsten Brants
• Keywords: ACL 2008, Computational Linguistics, Perfect Hash Function, Randomized Language Model, Standard N-gram Model