Predicting relative performance of classifiers from samples

This paper is concerned with the problem of predicting relative performance of classification algorithms. It focusses on methods that use results on small samples and discusses the shortcomings of previous approaches. A new variant is proposed that exploits, as some previous approaches, metalearning. The method requires that experiments be conducted on few samples. The information gathered is used to identify the nearest learning curve for which the sampling procedure was carried out fully. This in turn permits to generate a prediction regards the relative performance of algorithms. Experimental evaluation shows that the method competes well with previous approaches and provides quite good and practical solution to this problem.

• Type: Conference
• Year: 2005
• Where: ICML
• Authors: Rui Leite, Pavel Brazdil
• Keywords: Classification Algorithms, ICML 2005, Machine Learning, Nearest Learning Curve, Relative Performance