Ziv Lempel Compression of Huge Natural Language Data Tries Using Suffix Arrays

We present a very efficient, in terms of space and access speed, data structure for storing huge natural language data sets. The structure is described as LZ (Ziv Lempel) compressed linked list trie and is a step further beyond directed acyclic word graph in automata compression. We are using the structure to store DELAF, a huge French lexicon with syntactical, grammatical and lexical information associated with each word. The compressed structure can be produced in O(N) time using suffix trees for finding repetitions in trie, but for large data sets space requirements are more prohibitive than time so suffix arrays are used instead, with compression time complexity O(N log N) for all but for the largest data sets.

• Type: Conference
• Year: 1999
• Where: CPM
• Authors: Strahil Ristov, Eric Laporte
• Keywords: Combinatorics, CPM 1999, Data Sets, Directed Acyclic Word Graph, Language Data Sets