Optimal reverse prediction: a unified perspective on supervised, unsupervised and semi-supervised learning

Training principles for unsupervised learning are often derived from motivations that appear to be independent of supervised learning. In this paper we present a simple unification of several supervised and unsupervised training principles through the concept of optimal reverse prediction: predict the inputs from the target labels, optimizing both over model parameters and any missing labels. In particular, we show how supervised least squares, principal components analysis, k-means clustering and normalized graph-cut can all be expressed as instances of the same training principle. Natural forms of semisupervised regression and classification are then automatically derived, yielding semisupervised learning algorithms for regression and classification that, surprisingly, are novel and refine the state of the art. These algorithms can all be combined with standard regularizers and made non-linear via kernels.

• Type: Conference
• Year: 2009
• Where: ICML
• Authors: Linli Xu, Martha White, Dale Schuurmans
• Keywords: ICML 2009, Machine Learning, Semisupervised Learning Algorithms, Supervised Learning, Unsupervised Training Principles