Speech and sketching for multimodal design

While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. We studied the behavior of people sketching and speaking, and from this derived a set of rules for segmenting and aligning the signals from both modalities. Once the inputs are aligned, we use both modalities in interpretation. The result is a more natural interface to our system. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces—Natural language, Graphical user interfaces (GUI), Evaluation/methodology, Input devices and strategies (e.g., mouse, touchscreen), Interaction styles (e.g., commands, menus, forms, direct manipulation), User-centered design, Voice I/O General Terms Performance, Design, Experimentation, Human Factors Keywords speech, sketch, multimodal interaction

• Type: Conference
• Year: 2004
• Where: IUI
• Authors: Aaron Adler, Randall Davis
• Keywords: Early Stages, Graphical User Interfaces, Human Factors Keywords, IUI 2004, Software Engineering