Reduced n-gram Models for English and Chinese Corpora

Statistical language models should improve as the size of the n-grams increases from 3 to 5 or higher. However, the number of parameters and calculations, and the storage requirement increase very rapidly if we attempt to store all possible combinations of n-grams. To avoid these problems, the reduced n-grams' approach previously developed by O'Boyle (1993) can be applied. A reduced n-gram language model can store an entire corpus's phrase-history length within feasible storage limits. Another theoretical advantage of reduced n-grams is that they are closer to being semantically complete than traditional models, which include all n-grams. In our experiments, the reduced n-gram Zipf curves are first presented, and compared with previously obtained conventional n-grams for both English and Chinese. The reduced n-gram model is then applied to large English and Chinese corpora. For English, we can reduce the model sizes, compared to 7-gram traditional model sizes, with fact...

• Type: Conference
• Year: 2006
• Where: ACL
• Authors: Le Quan Ha, Philip Hanna, Darryl Stewart, F. Jack Smith
• Keywords: ACL 2006, ACL 2007, Language Model, N-grams Increases, Reduced N-gram Language