DEPEVAL(summ): Dependency-based Evaluation for Automatic Summaries

This paper presents DEPEVAL(summ), a dependency-based metric for automatic evaluation of summaries. Using a reranking parser and a Lexical-Functional Grammar (LFG) annotation, we produce a set of dependency triples for each summary. The dependency set for each candidate summary is then automatically compared against dependencies generated from model summaries. We examine a number of variations of the method, including the addition of WordNet, partial matching, or removing relation labels from the dependencies. In a test on TAC 2008 and DUC 2007 data, DEPEVAL(summ) achieves comparable or higher correlations with human judgments than the popular evaluation metrics ROUGE and Basic Elements (BE).

• Type: Journal
• Year: 2009
• Where: ACL
• Authors: Karolina Owczarzak
• Keywords: ACL 2009, Computational Linguistics, Dependency Triples, DUC 2007 Data, Popular Evaluation Metrics