Taking Account of the User's View in 3D Multimodal Instruction Dialogue

While recent advancements in virtual reality technology have created a rich communication interface linking humans and computers, there has been little work on building dialogue systems for 3D virtual worlds. This paper proposes a method for altering the instruction dialogue to match the user's view in a virtual environment. We illustrate the method with the system MID-3D, which interactively instructs the user on dismantling some parts of a car. First, in order to change the content of the instruction dialogue to match the user's view, we extend the re nement-driven planning algorithm by using the user's view as a plan constraint. Second, to manage the dialogue smoothly, the system keeps track of the user's viewpoint as part of the dialogue state and uses this information for coping with interruptive subdialogues. These mechanisms enable MID-3D to set instruction dialogues in an incremental way it takes account of the user's view even when it changes frequent...

• Type: Conference
• Year: 2000
• Where: COLING
• Authors: Yukiko I. Nakano, Kenji Imamura, Hisashi Ohara
• Keywords: Building Dialogue Systems, COLING 2000, COLING 2008, Instruction Dialogues, User's View