Learning to talk about events from narrated video in a construction grammar framework

The current research presents a system that learns to understand object names, spatial relation terms and event descriptions from observing narrated action sequences. The system extracts meaning from observed visual scenes by exploiting perceptual primitives related to motion and contact in order to represent events and spatial relations as predicate-argument structures. Learning the mapping between sentences and the predicate-argument representations of the situations they describe results in the development of a small lexicon, and a structured set of sentence form-to-meaning mappings, or simplified grammatical constructions. The acquired grammatical construction knowledge generalizes, allowing the system to correctly understand new sentences not used in training. In the context of discourse, the grammatical constructions are used in the inverse sense to generate sentences from meanings, allowing the system to describe visual scenes that it perceives. In question and answer dialogs w...

• Type: Journal
• Year: 2005
• Where: AI
• Authors: Peter Ford Dominey, Jean-David Boucher
• Keywords: AI 2005, Artificial Intelligence, Grammatical Constructions, Spatial Relations, Visual Scenes