Three new graphical models for statistical language modelling

The supremacy of n-gram models in statistical language modelling has recently been challenged by parametric models that use distributed representations to counteract the difficulties caused by data sparsity. We propose three new probabilistic language models that define the distribution of the next word in a sequence given several preceding words by using distributed representations of those words. We show how real-valued distributed representations for words can be learned at the same time as learning a large set of stochastic binary hidden features that are used to predict the distributed representation of the next word from previous distributed representations. Adding connections from the previous states of the binary hidden features improves performance as does adding direct connections between the real-valued distributed representations. One of our models significantly outperforms the very best ngram models.

• Type: Conference
• Year: 2007
• Where: ICML
• Authors: Andriy Mnih, Geoffrey E. Hinton
• Keywords: Binary Hidden Features, ICML 2007, Machine Learning, Probabilistic Language Models, Real-valued Distributed Representations