Warning: statistical benchmarking is addictive. Kicking the habit in machine learning

Algorithm performance evaluation is so entrenched in the Machine Learning community that one could call it an addiction. Like most addictions, it is harmful and very difficult to give up. It is harmful because it has serious limitations. Yet, we have great faith in it practicing it in a ritualistic manner: we follow a fixed set of rules telling us the measure, the data sets and the statistical test to use. When we read a paper, even as reviewers, we are not sufficiently critical of results that follow these rules. Here, we will debate what are the limitations and how to best address them. This paper may not cure the addiction but hopefully it will be a good first step along that road.

• Type: Journal
• Year: 2010
• Where: JETAI
• Authors: Chris Drummond, Nathalie Japkowicz
• Keywords: Addiction, Harmful, JETAI 2010, Machine Learning Community