Validating the web-based evaluation of NLG systems

The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internetbased approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.

• Type: Journal
• Year: 2009
• Where: ACL
• Authors: Alexander Koller, Kristina Striegnitz, Donna Byron, Justine Cassell, Robert Dale, Sara Dalzel-Job, Johanna D. Moore, Jon Oberlander
• Keywords: ACL 2009, Computational Linguistics, GIVE Challenge, NLG Evaluation Methodology, Nlg Systems