Annotating Students' Understanding of Science Concepts

This paper summarizes the annotation of fine-grained entailment relationships in the context of student answers to science assessment questions. We annotated a corpus of 15,357 answer pairs with 145,911 fine-grained entailment relationships. We provide the rationale forsuchfine-grainedanalysisanddiscussitsperceivedbenefitstoanIntelligentTutoringSystem. Thecorpusalsohaspotentialapplications in other areas, such as question answering and multi-document summarization. Annotators achieved 86.2% inter-annotator agreement (Kappa=0.728,correspondingtosubstantialagreement)annotatingthefine-grainedfacetsofreferenceanswerswithregardtounderstanding expressed in student answers and labeling from one of five possible detailed relationship categories. The corpus described in this paper, which is the only one providing such detailed entailment annotations, is available as a public resource for the research community. The corpus is expected to enable application development, not only for intelligent ...

• Type: Conference
• Year: 2008
• Where: LREC
• Authors: Rodney D. Nielsen, Wayne Ward, James Martin, Martha Palmer
• Keywords: Detailed Entailment Annotations, Education, Fine-grained Entailment Relationships, LREC 2008, Student Answers