Context based multimodal fusion

We present a generic approach to multimodal fusion which we call context based multimodal integration. Key to this approach is that every multimodal input event is interpreted and enriched with respect to its local turn context. This local turn context comprises all previously recognized input events and the dialogue state that both belong to the same user turn. We show that a production rule system is an elegant way to handle this context based multimodal integration and we describe a first implementation of the so-called PATE system. Finally, we present results from a first evaluation of this approach as part of a human-factors experiment with the COMIC system. Categories and Subject Descriptors H.5 [User Interfaces]: input devices and strategies, natural language, prototyping, theory and methods, user-centered design. General Terms Algorithms, Design, Human Factors Keywords Multimodal integration, fusion, speech and pen input, multimodal dialogue systems

• Type: Conference
• Year: 2004
• Where: ICMI
• Authors: Norbert Pfleger
• Keywords: ICMI 2004, Local Turn Context, Multimodal Input Event, Multimodal Integration