Period Disambiguation with Maxent Model

Abstract. This paper presents our recent work on period disambiguation, the kernel problem in sentence boundary identification, with the maximum entropy (Maxent) model. A number of experiments are conducted on PTB-II WSJ corpus for the investigation of how context window, feature space and lexical information such as abbreviated and sentence-initial words affect the learning performance. Such lexical information can be automatically acquired from a training corpus by a learner. Our experimental results show that extending the feature space to integrate these two kinds of lexical information can eliminate 93.52% of the remaining errors from the baseline Maxent model, achieving an F-score of 99.8227%.

• Type: Conference
• Year: 2005
• Where: IJCNLP
• Authors: Chunyu Kit, Xiaoyue Liu
• Keywords: Feature Space, IJCNLP 2005, Lexical Information, Natural Language Processing, PTB-II WSJ Corpus