Bundle Methods for Regularized Risk Minimization

A wide variety of machine learning problems can be described as minimizing a regularized risk functional, with different algorithms using different notions of risk and different regularizers. Examples include linear Support Vector Machines (SVMs), Gaussian Processes, Logistic Regression, Conditional Random Fields (CRFs), and Lasso amongst others. This paper describes the theory and implementation of a scalable and modular convex solver which solves all these estimation problems. It can be parallelized on a cluster of workstations, allows for data-locality, and can deal with regularizers such as L1 and L2 penalties. In addition to the unified framework we present tight convergence bounds, which show that our algorithm converges in O(1/ ) steps to precision for general convex problems and in O(log(1/ )) steps for continuously differentiable problems. We demonstrate the performance of our general purpose solver on a variety of publicly available datasets.

• Type: Journal
• Year: 2010
• Where: JMLR
• Authors: Choon Hui Teo, S. V. N. Vishwanathan, Alex J. Smola, Quoc V. Le
• Keywords: Conditional Random Fields, JMLR 2010, Linear Support Vector, Problems