Active Learning and the Total Cost of Annotation

Active learning (AL) promises to reduce the cost of annotating labeled datasets for trainable human language technologies. Contrary to expectations, when creating labeled training material for HPSG parse selection and later reusing it with other models, gains from AL may be negligible or even negative. This has serious implications for using AL, showing that additional cost-saving strategies may need to be adopted. We explore one such strategy: using a model during annotation to automate some of the decisions. Our best results show an 80% reduction in annotation cost compared with labeling randomly selected data with a single model.

• Type: Conference
• Year: 2004
• Where: EMNLP
• Authors: Jason Baldridge, Miles Osborne
• Keywords: Additional Cost-saving Strategies, EMNLP 2004, EMNLP 2007, HPSG Parse Selection, Trainable Human Language