Tokenization: Returning to a Long Solved Problem - A Survey, Contrastive Experiment, Recommendations, and Toolkit -

We examine some of the frequently disregarded subtleties of tokenization in Penn Treebank style, and present a new rule-based preprocessing toolkit that not only reproduces the Treebank tokenization with unmatched accuracy, but also maintains exact stand-off pointers to the original text and allows flexible configuration to diverse use cases (e.g. to genreor domain-specific idiosyncrasies).

• Type: Journal
• Year: 2012
• Where: ACL
• Authors: Rebecca Dridan, Stephan Oepen
• Keywords: ACL 2012, Computational Linguistics, Penn Treebank, Subtleties, Unmatched Accuracy