The Balanced Accuracy and Its Posterior Distribution

—Evaluating the performance of a classification algorithm critically requires a measure of the degree to which unseen examples have been identified with their correct class labels. In practice, generalizability is frequently estimated by averaging the accuracies obtained on individual crossvalidation folds. This procedure, however, is problematic in two ways. First, it does not allow for the derivation of meaningful confidence intervals. Second, it leads to an optimistic estimate when a biased classifier is tested on an imbalanced dataset. We show that both problems can be overcome by replacing the conventional point estimate of accuracy by an estimate of the posterior distribution of the balanced accuracy. Keywords-classification performance; generalizability; bias; class imbalance

• Type: Conference
• Year: 2010
• Where: ICPR
• Authors: Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, Joachim M. Buhmann
• Keywords: Computer Vision, Conventional Point Estimate, Correct Class Labels, ICPR 2010, Meaningful Confidence Intervals