Sensory Modality Segregation

Why are sensory modalities segregated the way they are? In this paper we show that sensory modalities are well designed for self-supervised cross-modal learning. Using the Minimizing-Disagreement algorithm on an unsupervised speech categorization task with visual (moving lips) and auditory (sound signal) inputs, we show that very informative auditory dimensions actually harm performance when moved to the visual side of the network. It is better to throw them away than to consider them part of the “visual input”. We explain this finding in terms of the statistical structure in sensory inputs.

• Type: Conference
• Year: 2003
• Where: NIPS
• Authors: Virginia R. de Sa
• Keywords: Informative Auditory Dimensions, NIPS 2003, NIPS 2007, Sensory Modalities, Unsupervised Speech Categorization