Training Continuous Space Language Models: Some Practical Issues

Using multi-layer neural networks to estimate the probabilities of word sequences is a promising research area in statistical language modeling, with applications in speech recognition and statistical machine translation. However, training such models for large vocabulary tasks is computationally challenging which does not scale easily to the huge corpora that are nowadays available. In this work, we study the performance and behavior of two neural statistical language models so as to highlight some important caveats of the classical training algorithms. The induced word embeddings for extreme cases are also analysed, thus providing insight into the convergence issues. A new initialization scheme and new training techniques are then introduced. These methods are shown to greatly reduce the training time and to significantly improve performance, both in terms of perplexity and on a large-scale translation task.

• Type: Journal
• Year: 2010
• Where: EMNLP
• Authors: Hai Son Le, Alexandre Allauzen, Guillaume Wisniewski, François Yvon
• Keywords: EMNLP 2010, Multi-layer Neural Networks, Natural Language Processing, Statistical Language, Statistical Language Modeling