A Connectionist Model of Anticipation in Visual Worlds

Recent “visual worlds” studies, wherein researchers study language in context by monitoring eye-movements in a visual scene during sentence processing, have revealed much about the interaction of diverse information sources and the time course of their influence on comprehension. In this study, five experiments that trade off scene context with a variety of linguistic factors are modelled with a Simple Recurrent Network modified to integrate a scene representation with the standard incremental input of a sentence. The results show that the model captures the qualitative behavior observed during the experiments, while retaining the ability to develop the correct interpretation in the absence of visual input.

• Type: Conference
• Year: 2005
• Where: IJCNLP
• Authors: Marshall R. Mayberry, Matthew W. Crocker, Pia Knoeferle
• Keywords: IJCNLP 2005, Natural Language Processing, Researchers Study Language, Standard Incremental Input, Visual Scene