Building a question answering test collection

The TREC-8 Question Answering (QA) Track was the first large-scale evaluation of domain-independent question answering systems. In addition to fostering research on the QA task, the track was used to investigate whether the evaluation methodology used for document retrieval is appropriate for a different natural language processing task. As with document relevance judging, assessors had legitimate differences of opinions as to whether a response actually answers a question, but comparative evaluation of QA systems was stable despite these differences. Creating a reusable QA test collection is fundamentally more difficult than creating a document retrieval test collection since the QA task has no equivalent to document identifiers.

• Type: Conference
• Year: 2000
• Where: SIGIR
• Authors: Ellen M. Voorhees, Dawn M. Tice
• Keywords: Document Retrieval, Information Management, QA Task, Question Answering, SIGIR 2000