An Empirical Evaluation of Automated Black Box Testing Techniques for Crashing GUIs

This paper reports an empirical evaluation of four blackbox testing techniques for crashing programs through their GUI interface: SH, AF, DH, and BxT. The techniques vary in their level of automation and the results they offer. The experiments we conducted quantify execution time and the capability of finding a crash for each technique on 8 different cellular phone configurations with historical (real) errors. The results show that AF and BxT offered better precision (i.e., the fraction of runs that end in a crash out of the total number of runs) than SH and DH (AF and BxT found crashes in all 8 configurations), and BxT crashes the application the fastest more often (5 out of 8 cases). The experiments reveal that the selection of the random seed to AF and BxT results in a high variance of execution time (i.e., the time the technique takes to either crash the application or timeout in 40h): the mean (across 8 phone configurations) of the standard deviation of execution times (for 10 ru...

• Type: Journal
• Year: 2009
• Where: ICST
• Authors: Cristiano Bertolini, Glaucia Peres, Marcelo d'Amorim, Alexandre Mota
• Keywords: Blackbox Testing Techniques, Execution Times, ICST 2009, Phone Configuration, Software Engineering