Enhanced Suffix Arrays as Language Models: Virtual k-Testable Languages

Abstract. In this article, we propose the use of suffix arrays to efficiently implement n-gram language models with practically unlimited size n. This approach, which is used with synchronous back-off, allows us to distinguish between alternative sequences using large contexts. We also show that we can build this kind of models with additional information for each symbol, such as part-of-speech tags and dependency information. The approach can also be viewed as a collection of virtual k-testable automata. Once built, we can directly access the results of any k-testable automaton generated from the input training data. Synchronous backoff automatically identifies the k-testable automaton with the largest feasible k. We have used this approach in several classification tasks.

• Type: Conference
• Year: 2010
• Where: ICGI
• Authors: Herman Stehouwer, Menno van Zaanen
• Keywords: ICGI 2010, K-testable Automaton, N-gram Language Models, Natural Language Processing, Suffix Arrays