Regularizing generalization error estimators: a novel approach to robust model selection

Abstract. A well-known result by Stein shows that regularized estimators with small bias often yield better estimates than unbiased estimators. In this paper, we adapt this spirit to model selection, and propose regularizing unbiased generalization error estimators for stabilization. We trade a small bias in a model selection criterion against a larger variance reduction which has the beneficial effect of being more precise on a single training set.

• Type: Conference
• Year: 2004
• Where: ESANN
• Authors: Masashi Sugiyama, Motoaki Kawanabe, Klaus-Robert Müller
• Keywords: ESANN 2004, ESANN 2007, Generalization Error Estimators, Small Bias, Unbiased Estimators