Exploring the use of crowdsourcing to support empirical studies in software engineering

The power and the generality of the findings obtained through empirical studies are bounded by the number and type of participating subjects. In software engineering, obtaining a large number of adequate subjects to evaluate a technique or tool is often a major challenge. In this work we explore the use of crowdsourcing as a mechanism to address that challenge by assisting in subject recruitment. More specifically, through this work we show how we adapted a study to be performed under an infrastructure that not only makes it possible to reach a large base of users but it also provides capabilities to manage those users as the study is being conducted. We discuss the lessons we learned through this experience, which illustrate the potential and tradeoffs of crowdsourcing software engineering studies. General Terms Experimentation, Measurement Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics Keywords empirical studies, crowdsourcing

• Type: Journal
• Year: 2010
• Where: ESEM
• Authors: Kathryn T. Stolee, Sebastian G. Elbaum
• Keywords: Empirical Studies, ESEM 2010, Keywords Empirical Studies, Software Engineering