A Framework for Identifying Textual Redundancy

The task of identifying redundant information in documents that are generated from multiple sources provides a significant challenge for summarization and QA systems. Traditional clustering techniques detect redundancy at the sentential level and do not guarantee the preservation of all information within the document. We discuss an algorithm that generates a novel graph-based representation for a document and then utilizes a set cover approximation algorithm to remove redundant text from it. Our experiments show that this approach offers a significant performance advantage over clustering when evaluated over an annotated dataset.

• Type: Conference
• Year: 2008
• Where: COLING
• Authors: Kapil Thadani, Kathleen McKeown
• Keywords: COLING 2008, Computational Linguistics, Cover Approximation Algorithm, Significant Performance Advantage, Traditional Clustering Techniques