Online Task Assignment in Crowdsourcing Markets

We explore the problem of assigning heterogeneous tasks to workers with different, unknown skill sets in crowdsourcing markets such as Amazon Mechanical Turk. We first formalize the online task assignment problem, in which a requester has a fixed set of tasks and a budget that specifies how many times he would like each task completed. Workers arrive one at a time (with the same worker potentially arriving multiple times), and must be assigned to a task upon arrival. The goal is to allocate workers to tasks in a way that maximizes the total benefit that the requester obtains from the completed work. Inspired by recent research on the online adwords problem, we present a two-phase exploration-exploitation assignment algorithm and prove that it is competitive with respect to the optimal offline algorithm which has access to the unknown skill levels of each worker. We empirically evaluate this algorithm using data collected on Mechanical Turk and show that it performs better than ra...

• Type: Journal
• Year: 2012
• Where: AAAI
• Authors: Chien-Ju Ho, Jennifer Wortman Vaughan
• Keywords: AAAI 2012, Amazon Mechanical Turk, Empirical Validation, Greedy Algorithms, Intelligent Agents