An Evaluation Framework for Semantic Search in P2P Networks

: We address the problem of evaluating peer-to-peer information retrieval (P2PIR) systems with semantic overlay structure. The P2PIR community lacks a commonly accepted testbed, such as TREC is for the classic IR community. The problem with using classic test collections in a P2P scenario is that they provide no realistic distribution of documents and queries over peers, which is, however, crucial for realistically simulating and evaluating semantic overlays. On the other hand, document collections that can be easily distributed (e.g. by exploiting categories or author information) lack both queries and relevance judgments. Therefore, we propose an evaluation framework, which provides a strategy for constructing a P2PIR testbed, consisting of a prescription for content distribution, query generation and measuring effectiveness without the need for human relevance judgments. It can be used with any document collection that contains author information and document relatedness (e.g. refer...

• Type: Conference
• Year: 2010
• Where: IICS
• Authors: Florian Holz, Hans Friedrich Witschel, Gregor Heinrich, Gerhard Heyer, Sven Teresniak
• Keywords: Author Information, IICS 2010, Internet Technology, Relevance Judgments, Semantic Overlays