Detecting communication errors from visual cues during the system's conversational turn

Automatic detection of communication errors in conversational systems has been explored extensively in the speech community. However, most previous studies have used only acoustic cues. Visual information has also been used by the speech community to improve speech recognition in dialogue systems, but this visual information is only used when the speaker is communicating vocally. A recent perceptual study indicated that human observers can detect communication problems when they see the visual footage of the speaker during the system’s reply. In this paper, we present work in progress towards the development of a communication error detector that exploits this visual cue. In datasets we collected or acquired, facial motion features and head poses were estimated while users were listening to the system response and passed to a classifier for detecting a communication error. Preliminary experiments have demonstrated that the speaker’s visual information during the system’s reply ...

• Type: Conference
• Year: 2007
• Where: ICMI
• Authors: Sy Bor Wang, David Demirdjian, Trevor Darrell
• Keywords: Communication Error, ICMI 2007, Speech Community, Visual Information