Examining the Robustness of Evaluation Metrics for Patent Retrieval with Incomplete Relevance Judgements

Recent years have seen a growing interest in research into patent retrieval. One of the key issues in conducting information retrieval (IR) research is meaningful evaluation of the effectiveness of the retrieval techniques applied to task under investigation. Unlike many existing well explored IR tasks where the focus is on achieving high retrieval precision, patent retrieval is to a significant degree a recall focused task. The standard evaluation metric used for patent retrieval evaluation tasks is currently mean average precision (MAP). However this does not reflect system recall well. Meanwhile, the alternative of using the standard recall measure does not reflect user search effort, which is a significant factor in practical patent search environments. In recent work we introduce a novel evaluation metric for patent retrieval evaluation (PRES) [13]. This is designed to reflect both system recall and user effort. Analysis of PRES demonstrated its greater effectiveness in evaluating...

• Type: Journal
• Year: 2010
• Where: CLEF
• Authors: Walid Magdy, Gareth J. F. Jones
• Keywords: CLEF 2010, Evaluation, Information Technology, Patent Retrieval, Patent Retrieval Evaluation