Finding Faults: Manual Testing vs. Random+ Testing vs. User Reports

The usual way to compare testing strategies, whether theoretically or empirically, is to compare the number of faults they detect. To ascertain definitely that a testing strategy is better than another, this is a rather coarse criterion: shouldn’t the nature of faults matter as well as their number? The empirical study reported here confirms this conjecture. An analysis of faults detected in Eiffel libraries through three different techniques—random tests, manual tests, and user incident reports—shows that each is good at uncovering significantly different kinds of faults. None of the techniques subsumes any of the others, but each brings distinct contributions.

• Type: Conference
• Year: 2008
• Where: ISSRE
• Authors: Ilinca Ciupa, Bertrand Meyer, Manuel Oriol, Alexander Pretschner
• Keywords: Faults, ISSRE 2008, Software Engineering, Testing Strategies, Usual Way