Language and Task Independent Text Categorization with Simple Language Models

We present a simple method for language independent and task independent text categorization learning, based on character-level n-gram language models. Our approach uses simple information theoretic principles and achieves effective performance across a variety of languages and tasks without requiring feature selection or extensive pre-processing. To demonstrate the language and task independence of the proposed technique, we present experimental results on several languages—Greek, English, Chinese and Japanese—in several text categorization problems—language identification, authorship attribution, text genre classification, and topic detection. Our experimental results show that the simple approach achieves state of the art performance in each case.

• Type: Conference
• Year: 2003
• Where: NAACL
• Authors: Fuchun Peng, Dale Schuurmans, Shaojun Wang
• Keywords: Character-level N-gram Language, Independent Text Categorization, NAACL 2003, NAACL 2007, Text Categorization