Focused web crawling in the acquisition of comparable corpora

CLIR resources, such as dictionaries and parallel corpora, are scarce for special domains. Obtaining comparable corpora automatically for such domains could be an answer to this problem. The Web, with its vast volumes of data, offers a natural source for this. We experimented with focused crawling as a means to acquire comparable corpora in the genomics domain. The acquired corpora were used to statistically translate domainspecific words. The same words were also translated using a high-quality, but non-genomics-related parallel corpus, which fared considerably worse. We also evaluated our system with standard IR experiments, combining statistical translation using the Web corpora with dictionary-based translation. The results showed improvement over pure dictionary-based translation. Therefore, mining the Web for comparable corpora seems promising.

• Type: Journal
• Year: 2008
• Where: IR
• Authors: Tuomas Talvensaari, Ari Pirkola, Kalervo Järvelin, Martti Juhola, Jorma Laurikkala
• Keywords: CLIR Resources, Comparable Corpora, Dictionary-based Translation, IR 2008, Natural Language Processing