Choosing Learning Algorithms Using Sign Tests with High Replicability

An important task in machine learning is determining which learning algorithm works best for a given data set. When the amount of data is small the same data needs to be used repeatedly in order to get a reasonable estimate of the accuracy of the learning algorithms. This results in violations of assumptions on which standard tests are based and makes it hard to design a good test. In this article, we investigate sign tests to address the problem of choosing the best of two learning algorithms when only a small data set is available. Sign tests are conceptually simple and no assumption about underlying distributions is required. We show that simplistic sample generation can lead to flawed test outcomes. Furthermore, we identify a test that performs well based on Type I error (showing a difference between algorithms when there is none), power (showing a difference when it indeed exists) and replicability. Replicability is a novel measure of a quality of a test that gives an indicatio...

• Type: Conference
• Year: 2003
• Where: AUSAI
• Authors: Remco R. Bouckaert
• Keywords: Artificial Intelligence, AUSAI 2003, Data Set, Learning Algorithms, Sign Tests