Unsupervised Segmentation of Chinese Text by Use of Branching Entropy

We propose an unsupervised segmentation method based on an assumption about language data: that the increasing point of entropy of successive characters is the location of a word boundary. A large-scale experiment was conducted by using 200 MB of unsegmented training data and 1 MB of test data, and precision of 90% wasattained with recall being around 80%. Moreover, we found that the precision was stable at around 90% independently of the learning data size.

• Type: Conference
• Year: 2006
• Where: ACL
• Authors: Zhihui Jin, Kumiko Tanaka-Ishii
• Keywords: ACL 2006, ACL 2007, Language Data, Unsegmented Training Data, Unsupervised Segmentation Method