Overlapping experiment infrastructure: more, better, faster experimentation

At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments, how to run experiments that produce better decisions, and how to run them faster. In this paper, we describe Google’s overlapping experiment infrastructure that is a key component to solving these problems. In addition, because an experiment infrastructure alone is insufficient, we also discuss the associated tools and educational processes required to use it effectively. We conclude by describing trends that show the success of this overall experimental environment. While the paper specifically describes the experime...

• Type: Conference
• Year: 2010
• Where: KDD
• Authors: Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike Meyer
• Keywords: A/B Testing, Data Mining, KDD 2010, Machine Learning Algorithms, Obvious User-visible Changes