On evaluation of adaptive topic tracking systems

Summative evaluation methods for supervised adaptive topic tracking systems convolve the effect of system decisions on present utility with the effect on future utility. This paper describes a new formative evaluation approach that focuses on future utility for use in the design stage of adaptive systems. Topic model quality is assessed at a predefined set of points using a fixed document set to enhance comparability. Experiments using a vector-space topic tracking system illustrate the utility of this approach to formative evaluation. Categories and Subject Descriptors: H.3.4 [Information Storage and Retrieval]: Systems and Software – Performance Evaluation. General Terms: Design, Measurement, Performance.

• Type: Conference
• Year: 2005
• Where: SIGIR
• Authors: Tamer Elsayed, Douglas W. Oard
• Keywords: Formative Evaluation, Future Utility, SIGIR 2005, Topic Tracking