A Comparison of Document, Sentence, and Term Event Spaces

The trend in information retrieval systems is from document to sub-document retrieval, such as sentences in a summarization system and words or phrases in question-answering system. Despite this trend, systems continue to model language at a document level using the inverse document frequency (IDF). In this paper, we compare and contrast IDF with inverse sentence frequency (ISF) and inverse term frequency (ITF). A direct comparison reveals that all language models are highly correlated; however, the average ISF and ITF values are 5.5 and 10.4 higher than IDF. All language models appeared to follow a power law distribution with a slope coefficient of

• Type: Conference
• Year: 2006
• Where: ACL
• Authors: Catherine Blake
• Keywords: ACL 2006, ACL 2007, Inverse Document Frequency, Inverse Sentence Frequency, Language Models