Evaluating Robustness Of A QA System Through A Corpus Of Real-Life Questions

This paper presents the sequential evaluation of the question answering system SQuaLIA. This system is based on the same sequential process as most statistical question answering systems, involving 4 main steps from question analysis to answer extraction.The evaluation is based on a corpus made from 20 questions taken in the set of an evaluation campaign and which were well answered by SQuaLIA. Each of the 20 questions have been typed by 17 participants natives, non natives and dyslexics. They were vocally instructed the target of each question. Each of the 4 analysis step of the system involves a loss of accuracy, until an average of 60% of right answers at the end of the process. The main cause of this loss seem to be the orthographic mistakes users make on nouns. The most natural way for users to query search engines for specific information is to type a question. From this natural interaction the interest for efficient question raises Many evaluation campaigns on question answerin...

• Type: Conference
• Year: 2008
• Where: LREC
• Authors: Laurianne Sitbon, Patrice Bellot, Philippe Blache
• Keywords: Education, Evaluation, LREC 2008, Question Answering, Question Answering Systems