Current Developments in Information Retrieval Evaluation

: In the last decade, many evaluation results have been created within the evaluation initiatives like TREC, NTCIR and CLEF. The large amount of data available has led to substantial research on the validity of the evaluation procedure. An evaluation based on the Cranfield paradigm requires basically topics as descriptions of information needs, a document collection, systems to compare, human jurors to judge the documents retrieved by the systems against the information needs descriptions and some metric to compare the systems. For all these elements, there has been a scientific discussion. How many topics, systems, jurors and juror decisions are necessary to achieve valid results? How can the validity be measured? Which metrics are the most reliable ones and which metrics are appropriate from a user perspective? Examples from current CLEF experiments are used to illustrate some of the issues. User based evaluations confront test users with the results of search systems and let them so...

• Type: Conference
• Year: 2009
• Where: ECIR
• Authors: Thomas Mandl
• Keywords: Computer Science, ECIR 2009, Relevant Documents, User Based Evaluations, User Satisfaction