Can we do without GUIs? Gesture and speech interaction with a patient information system

We have developed a gesture input system that provides a common interaction technique across mobile, wearable and ubiquitous computing devices of diverse form factors. In this paper, we combine our gestural input technique with speech output and test whether or not the absence of a visual display impairs usability in this kind of multimodal interaction. This is of particular relevance to mobile, wearable and ubiquitous systems where visual displays may be restricted or unavailable. We conducted the evaluation using a prototype for a system combining gesture input and speech output to provide information to patients in a hospital Accident and Emergency department. A group of participants was instructed to access various services using gestural inputs. The services were delivered by automated speech output. Throughout their tasks, these participants could see a visual display on which a GUI presented the available services and their corresponding gestures. Another group of participants ...

• Type: Journal
• Year: 2006
• Where: PUC
• Authors: Eamonn O'Neill, Manasawee Kaenampornpan, Vassilis Kostakos, Andrew Warr, Dawn Woodgate
• Keywords: Gesture, Gesture Input, PUC 2006, Visual Display