User studies and the design of Natural Language Systems

This paper presents a critical discussion of the various approaches that have been used in the evaluation of Natural Language systems. We conclude that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identifyNL requirements for carrying out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual reference and reference to the structure of the information source. We discuss how these might be satisfied by future Natural Language systems.

• Type: Conference
• Year: 1989
• Where: EACL
• Authors: Steve Whittaker, Phil Stenton
• Keywords: EACL 1989, Natural Language Processing, Natural Language Systems, Prototype Natural Language, Task Requiring Data