Suboptimal Behavior of Bayes and MDL in Classification Under Misspecification

We show that forms of Bayesian and MDL inference that are often applied to classification problems can be inconsistent. This means that there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. From a Bayesian point of view, the result can be reinterpreted as saying that Bayesian inference can be inconsistent under misspecification, even for countably infinite models. We extensively discuss the result from both a Bayesian and an MDL perspective. Keywords Bayesian statistics . Minimum description length . Classification . Consistency . Inconsistency . Misspecification

• Type: Conference
• Year: 2004
• Where: COLT
• Authors: Peter Grünwald, John Langford
• Keywords: COLT 2004, Generalization Error, Machine Learning, MDL Inference, Smallest Achievable Generalization