Sentiment Summarization: Evaluating and Learning User Preferences

We present the results of a large-scale, end-to-end human evaluation of various sentiment summarization models. The evaluation shows that users have a strong preference for summarizers that model sentiment over non-sentiment baselines, but have no broad overall preference between any of the sentiment-based models. However, an analysis of the human judgments suggests that there are identifiable situations where one summarizer is generally preferred over the others. We exploit this fact to build a new summarizer by training a ranking SVM model over the set of human preference judgments that were collected during the evaluation, which results in a 30% relative reduction in error over the previous best summarizer.

• Type: Conference
• Year: 2009
• Where: EACL
• Authors: Kevin Lerman, Sasha Blair-Goldensohn, Ryan T. McDonald
• Keywords: Broad Overall Preference, EACL 2009, End-to-end Human Evaluation, Human Preference Judgments, Natural Language Processing