A Lightweight and Efficient Tool for Cleaning Web Pages

Originally conceived as a "naive" baseline experiment using traditional n-gram language models as classifiers, the NCLEANER system has turned out to be a fast and lightweight tool for cleaning Web pages with state-of-the-art accuracy (based on results from the CLEANEVAL competition held in 2007). Despite its simplicity, the algorithm achieves a significant improvement over the baseline (i.e. plain, uncleaned text dumps), trading off recall for substantially higher precision. NCLEANER is available as an open-source software package. It is preconfigured for English Web pages, but can adapted to other languages with minimal amounts of manually cleaned training data. Since NCLEANER does not make use of HTML structure, it can also be applied to existing Web corpora that are only available in plain text format, with a minor loss in classfication accuracy.

• Type: Conference
• Year: 2008
• Where: LREC
• Authors: Stefan Evert
• Keywords: Education, LREC 2008, N-gram Language Models, Uncleaned Text Dumps, Web Pages