They Can Help: Using Crowdsourcing to Improve the Evaluation of Grammatical Error Detection Systems

Despite the rising interest in developing grammatical error detection systems for non-native speakers of English, progress in the field has been hampered by a lack of informative metrics and an inability to directly compare the performance of systems developed by different researchers. In this paper we address these problems by presenting two evaluation methodologies, both based on a novel use of crowdsourcing. 1 Motivation and Contributions One of the fastest growing areas in need of NLP tools is the field of grammatical error detection for learners of English as a Second Language (ESL). According to Guo and Beckett (2007), “over a billion people speak English as their second or foreign language.” This high demand has resulted in many NLP research papers on the topic, a Synthesis Series book (Leacock et al., 2010) and a recurring workshop (Tetreault et al., 2010a), all in the last five years. In this year’s ACL conference, there are four long papers devoted to this topic. De...

• Type: Journal
• Year: 2011
• Where: ACL
• Authors: Nitin Madnani, Martin Chodorow, Joel R. Tetreault, Alla Rozovskaya
• Keywords: ACL 2011, Computational Linguistics, Evaluation Methodologies, Language Esl, Synthesis Series