Shepherding the crowd yields better work

Micro-task platforms provide massively parallel, ondemand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, taskspecific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participan...

• Type: Journal
• Year: 2012
• Where: CSCW
• Authors: Steven Dow, Anand Pramod Kulkarni, Scott R. Klemmer, Björn Hartmann
• Keywords: CSCW 2012, Expert Feedback, Information Interfaces, Social Sciences, Work Participants