Evaluating Evaluation Methods for Generation in the Presence of Variation

Recent years have seen increasing interest in automatic metrics for the evaluation of generation systems. When a system can generate syntactic variation, automatic evaluation becomes more difficult. In this paper, we compare the performance of several automatic evaluation metrics using a corpus of automatically generated paraphrases. We show that these evaluation metrics can at least partially measure adequacy (similarity in meaning), but are not good measures of fluency (syntactic correctness). We make several proposals for improving the evaluation of generation systems that produce variation.

• Type: Conference
• Year: 2005
• Where: CICLING
• Authors: Amanda Stent, Matthew Marge, Mohit Singhai
• Keywords: Automatic Evaluation, CICLING 2005, Evaluation, Evaluation Metrics, Natural Language Processing