Large Scale Parallel Document Mining for Machine Translation

A distributed system is described that reliably mines parallel text from large corpora. The approach can be regarded as cross-language near-duplicate detection, enabled by an initial, low-quality batch translation. In contrast to other approaches which require specialized metadata, the system uses only the textual content of the documents. Results are presented for a corpus of over two billion web pages and for a large collection of digitized public-domain books.

• Type: Journal
• Year: 2010
• Where: COLING
• Authors: Jakob Uszkoreit, Jay Ponte, Ashok C. Popat, Moshe Dubiner
• Keywords: COLING 2010, Computational Linguistics, Large Corpora, Low-quality Batch Translation, Parallel Text