Using self-supervised word segmentation in Chinese information retrieval

We propose a self-supervised word-segmentation technique for Chinese information retrieval. This method combines the advantages of traditional dictionary based approaches with character based approaches, while overcoming many of their shortcomings. Experiments on TREC data show comparable performance to both the dictionary based and the character based approaches. However, our method is language independent and unsupervised, which provides a promising avenue for constructing accurate multilingual information retrieval systems that are flexible and adaptive. Categories and Subject Descriptors I.2 [Computing Methodologies]: artificial intelligence General Terms Experimentation Keywords Self-supervised word segmentation, Chinese IR

• Type: Journal
• Year: 2002
• Where: SIGIR
• Authors: Fuchun Peng, Xiangji Huang, Dale Schuurmans, Nick Cercone, Stephen E. Robertson
• Keywords: Character Based Approaches, Dictionary Based Approaches, Information Retrieval, Information Technology, SIGIR 2002