A Critical Assessment of Benchmark Comparison in Planning

Recent trends in planning research have led to empirical comparison becoming commonplace. The eld has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1 the performance of a general purpose planner should not be penalized biased if executed on a sampling of problems and domains, PR2 minor syntactic di erences in representation do not a ect performance, and PR3 problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1 the latest version of a planner is the best one to use, PL2 default parameter settings approximate good performance, and PL3 time cut-o s do not unduly bias outcome. The metrics assumptions are: M1 performance degrades si...

• Type: Journal
• Year: 2002
• Where: JAIR
• Authors: Adele E. Howe, Eric Dahlman
• Keywords: JAIR 2002, Obvious Practical Reasons, Performance, Problems