Modeling Annotators: A Generative Approach to Learning from Annotator Rationales

A human annotator can provide hints to a machine learner by highlighting contextual "rationales" for each of his or her annotations (Zaidan et al., 2007). How can one exploit this side information to better learn the desired parameters ? We present a generative model of how a given annotator, knowing the true , stochastically chooses rationales. Thus, observing the rationales helps us infer the true . We collect substring rationales for a sentiment classification task (Pang and Lee, 2004) and use them to obtain significant accuracy improvements for each annotator. Our new generative approach exploits the rationales more effectively than our previous "masking SVM" approach. It is also more principled, and could be adapted to help learn other kinds of probabilistic classifiers for quite different tasks. 1 Background Many recent papers aim to reduce the amount of annotated data needed to train the parameters of a statistical model. Well-known paradigms include active ...

• Type: Conference
• Year: 2008
• Where: EMNLP
• Authors: Omar Zaidan, Jason Eisner
• Keywords: EMNLP 2008, Human Annotators, Natural Language Processing, Sentiment Classification Task, Significant Accuracy Improvements