Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators

In the field of machine translation, automatic metrics have proven quite valuable in system development for tracking progress and measuring the impact of incremental changes. However, human judgment still plays a large role in the context of evaluating MT systems. For example, the GALE project uses humantargeted translation edit rate (HTER), wherein the MT output is scored against a post-edited version of itself (as opposed to being scored against an existing human reference). This poses a problem for MT researchers, since HTER is not an easy metric to calculate, and would require hiring and training human annotators to perform the editing task. In this work, we explore soliciting those edits from untrained human annotators, via the online service Amazon Mechanical Turk. We show that the collected data allows us to predict HTER-ranking of documents at a significantly higher level than the ranking obtained using automatic metrics.

• Type: Journal
• Year: 2010
• Where: NAACL
• Authors: Omar Zaidan, Chris Callison-Burch
• Keywords: Automatic Metrics, Computational Linguistics, Human, Human Annotators, NAACL 2010