Extension of Zipf's Law to Words and Phrases

Zipf's law states that the frequency of word tokens in a large corpus of natural language is inversely proportional to the rank. The law is investigated for two languages English and Mandarin and for ngram word phrases as well as for single words. The law for single words is shown to be valid only for high frequency words. However, when single word and n-gram phrases are combined together in one list and put in order of frequency the combined list follows Zipf's law accurately for all words and phrases, down to the lowest frequencies in both languages. The Zipf curves for the two languages are then almost identical.

• Type: Journal
• Year: 2002
• Where: COLING
• Authors: Le Quan Ha, Elvira I. Sicilia-Garcia, Ji Ming, F. Jack Smith
• Keywords: COLING 2002, COLING 2008, Inversely Proportional, Single Word, Word Tokens