Practical guide to controlled experiments on the web: listen to your customers not to the hippo

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments (single-factor or factorial designs), A/B tests (and their generalizations), split tests, Control/Treatment tests, and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-oninvestment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas tha...

• Type: Conference
• Year: 2007
• Where: KDD
• Authors: Ron Kohavi, Randal M. Henne, Dan Sommerfield
• Keywords: Data Mining, Data Mining Techniques, Descriptors G.3 Probability, Extensive Practical Experience, KDD 2007