Decision-Theoretic Control of Crowd-Sourced Workflows

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people ("workers") as an open call (e.g., on Amazon's Mechanical Turk). Crowd-sourcing has become immensely popular with hoards of employers ("requesters"), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized subtasks that are combined into a complex, iterative workflow in which workers check and improve each other's results. This paper raises an exciting question for AI -- could an autonomous agent control these workflows without human intervention, yielding better results than today's state of the art, a fixed control program? We describe a planner, TURKONTROL, that formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifac...

• Type: Conference
• Year: 2010
• Where: AAAI
• Authors: Peng Dai, Mausam, Daniel S. Weld
• Keywords: AAAI 2010, Amazon's Mechanical Turk, Intelligent Agents, Workflow, Workflow Control