TurKit: Tools for iterative tasks on mechanical turk

Mechanical Turk (MTurk) is an increasingly popular web service for paying people small rewards to do human computation tasks. Current uses of MTurk typically post independent parallel tasks. This paper explores an alternative iterative paradigm, in which workers build on or evaluate each other’s work. We describe TurKit, a new toolkit for deploying iterative tasks to MTurk, with a familiar imperative programming paradigm that effectively uses MTurk workers as subroutines, such as the comparison function of a sorting algorithm. The toolkit handles the latency of MTurk tasks (typically measured in minutes), supports parallel tasks, and provides fault tolerance to avoid wasting money and time. We present a variety of iterative experiments using TurKit, including image description, copy editing, handwriting recognition, and sorting. ACM Classification: H5.2 [Information interfaces and presentation]: User Interfaces. - Prototyping. General terms: Algorithms, Design, Economics, Experiment...

• Type: Conference
• Year: 2009
• Where: VL
• Authors: Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller
• Keywords: Human Computation Tasks, Independent Parallel Tasks, Languages, Parallel Tasks, VL 2009