The Power of Amnesia

We propose a learning algorithm for a variable memory length Markov process. Human communication, whether given as text, handwriting, or speech, has multi characteristic time scales. On short scales it is characterized mostly by the dynamics that generate the process, whereas on large scales, more syntactic and semantic information is carried. For that reason the conventionally used xed memory Markov models cannot capture e ectively the complexity of such structures. On the other hand using long memory models uniformly is not practical even for as short memory as four. The algorithm we propose is based on minimizing the statistical prediction error by extending the memory, or state length, adaptively, until the total prediction error is su ciently small. We demonstrate the algorithmby learning the structure of natural English text and applying the learned model to the correction of corrupted text. Using less than 3000 states the model's performance is far superior to that of xed ...

• Type: Conference
• Year: 1993
• Where: NIPS
• Authors: Dana Ron, Yoram Singer, Naftali Tishby
• Keywords: Memory Length Markov, Memory Markov Models, Memory Models, NIPS 1993, NIPS 2007