Distributional Semantics in Technicolor

Our research aims at building computational models of word meaning that are perceptually grounded. Using computer vision techniques, we build visual and multimodal distributional models and compare them to standard textual models. Our results show that, while visual models with state-of-the-art computer vision techniques perform worse than textual models in general tasks (accounting for semantic relatedness), they are as good or better models of the meaning of words with visual correlates such as color terms, even in a nontrivial task that involves nonliteral uses of such words. Moreover, we show that visual and textual information are tapping on different aspects of meaning, and indeed combining them in multimodal models often improves performance.

• Type: Journal
• Year: 2012
• Where: ACL
• Authors: Elia Bruni, Gemma Boleda, Marco Baroni, Nam-Khanh Tran
• Keywords: ACL 2012, Computational Linguistics, Computer Vision Techniques, Nontrivial Task, Visual Models