Objective and automated protocols for the evaluation of biomedical search engines using No Title Evaluation protocols

Background: The evaluation of information retrieval techniques has traditionally relied on human judges to determine which documents are relevant to a query and which are not. This protocol is used in the Text Retrieval Evaluation Conference (TREC), organized annually for the past 15 years, to support the unbiased evaluation of novel information retrieval approaches. The TREC Genomics Track has recently been introduced to measure the performance of information retrieval for biomedical applications. Results: We describe two protocols for evaluating biomedical information retrieval techniques without human relevance judgments. We call these protocols No Title Evaluation (NT Evaluation). The first protocol measures performance for focused searches, where only one relevant document exists for each query. The second protocol measures performance for queries expected to have potentially many relevant documents per query (high-recall searches). Both protocols take e of the clear separation o...

• Type: Journal
• Year: 2008
• Where: BMCBI
• Authors: Fabien Campagne
• Keywords: BMCBI 2008, Evaluation, Information Retrieval, NT Evaluation