Older versions of the ROUGEeval summarization evaluation system were easier to fool

We show some limitations of the ROUGE evaluation method for automatic summarization. We present a method for automatic summarization based on a Markov model of the source text. By a simple greedy word selection strategy, summaries with high ROUGE-scores are generated. These summaries would however not be considered good by human readers. The method can be adapted to trick different settings of the ROUGEeval package. Key words: automatic summarization, automatic evaluation, Markov models

• Type: Journal
• Year: 2007
• Where: IPM
• Authors: Jonas Sjöbergh
• Keywords: Automatic Summarization, IPM 2007, Markov Models, ROUGE Evaluation Method