Replication of defect prediction studies: problems, pitfalls and recommendations

Background: The main goal of the PROMISE repository is to enable reproducible, and thus verifiable or refutable research. Over time, plenty of data sets became available, especially for defect prediction problems. Aims: In this study, we investigate possible problems and pitfalls that occur during replication. This information can be used for future replication studies, and serve as a guideline for researchers reporting novel results. Method: We replicate two recent defect prediction studies comparing different data sets and learning algorithms, and report missing information and problems. Results: Even with access to the original data sets, replicating previous studies may not lead to the exact same results. The choice of evaluation procedures, performance measures and presentation has a large influence on the reproducibility. Additionally, we show that trivial and random models can be used to identify overly optimistic evaluation measures. Conclusions: The best way to conduct easily...

• Type: Journal
• Year: 2010
• Where: PROMISE
• Authors: Thilo Mende
• Keywords: Data Sets, Defect Prediction, Performance Measures, PROMISE 2010, Software Engineering