Advanced Capabilities for Evaluating Student Writing: Detecting Off-Topic Essays Without Topic-Specific Training

We have developed a method to identify when a student essay is off-topic, i.e. the essay does not respond to the test question topic. This task is motivated by a real-world problem: detecting when students using a commercial essay evaluation system, CriterionSM , enter off-topic essays. Sometimes this is done in bad faith to trick the system; other times it is inadvertent, and the student has cut-and-pasted the wrong selection into the system. All previous methods that perform this task require 200-300 human scored essays for training purposes. However, there are situations in which no essays are available for training, such as when a user (teacher) wants to spontaneously write a new topic for her students. For these kinds of cases, we need a system that works reliably without training data. This paper describes an algorithm that detects when a student’s essay is offtopic without requiring a set of topic-specific essays for training. The system also distinguishes between two differen...

• Type: Conference
• Year: 2005
• Where: AIED
• Authors: Jill Burstein, Derrick Higgins
• Keywords: AIED 2005, Artificial Intelligence, Enter Off-topic Essays, Essays, Off-topic Writing