Evaluation Challenges in Large-Scale Document Summarization

We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than anual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and drawbacks of all evaluation methods and how they rank the different summarizers.

• Type: Conference
• Year: 2003
• Where: ACL
• Authors: Dragomir R. Radev, Simone Teufel, Horacio Saggion, Wai Lam, John Blitzer, Hong Qi, Arda Çelebi, Danyu Liu, Elliott Drábek
• Keywords: ACL 2003, ACL 2007, Evaluation Measures, Large-scale Meta Evaluation, Multi-document Summarizers