Identification of Document Language is Not yet a Completely Solved Problem

Existing Language Identification (LID) approaches do reach 100% precision, in most common situations, when dealing with documents written in just one language, and when those documents are large enough. However, LID approaches do not provide a reliable solution for some situations: when there is need to discriminate the correct variant of the language used in a text, for example, European or Brazilian variants of Portuguese, UK or USA English variants, or any other language variants. Another hard context occur with small touristic advertisements on the web, addressing foreigners but using local language to name most local entities. In this paper, we present a fully statisticsbased LID approach which learns the most discriminant information according to each context, and identifies the correct language or language variant a text is written in. This methodology is shown to be correct for normal texts and maintains its robustness in hard LID contexts.

• Type: Conference
• Year: 2006
• Where: CIMCA
• Authors: Joaquim Ferreira da Silva, Gabriel Pereira Lopes
• Keywords: CIMCA 2006, Intelligent Agents, Language Identification, Language Variant, LID Approaches