A Statistical Corpus-Based Term Extractor

Abstract. Term extraction is an important problem in natural language processing. In this paper, we propose a language independent statistical corpus-based term extraction algorithm. In previous approaches, evaluation has been subjective, at best relying on a lexicographer’s judgement. We evaluate the quality of our term extractor by assessing its predictiveness on an unseen corpus using perplexity. Second, we evaluate the precision and recall of our extractor by comparing the Chinese words in a segmented corpus with the words extracted by our system.

• Type: Conference
• Year: 2001
• Where: AI
• Authors: Patrick Pantel, Dekang Lin
• Keywords: AI 2001, Artificial Intelligence, Corpus-based Term Extraction, Natural Language Processing, Term Extraction