Towards maximizing the accuracy of human-labeled sensor data

We present two studies that evaluate the accuracy of human responses to an intelligent agent’s data classification questions. Prior work has shown that agents can elicit accurate human responses, but the applications vary widely in the data features and prediction information they provide to the labelers when asking for help. In an initial analysis of this work, we found the five most popular features, namely uncertainty, amount and level of context, prediction of an answer, and request for user feedback. We propose that there is a set of these data features and prediction information that maximizes the accuracy of labeler responses. In our first study, we compare accuracy of users of an activity recognizer labeling their own data across the dimensions. In the second study, participants were asked to classify a stranger’s emails into folders and strangers’ work activities by interruptibility. We compared the accuracy of the responses to the users’ self-reports across the same ...

• Type: Conference
• Year: 2010
• Where: IUI
• Authors: Stephanie Rosenthal, Anind K. Dey
• Keywords: Accurate Human Responses, Accurate Responses, IUI 2010, Labeler Responses, Software Engineering