Accelerating Newton Optimization for Log-Linear Models through Feature Redundancy

— Log-linear models are widely used for labeling feature vectors and graphical models, typically to estimate robust conditional distributions in presence of a large number of potentially redundant features. Limited-memory quasi-Newton methods like LBFGS or BLMVM are optimization workhorses for such applications, and most of the training time is spent computing the objective and gradient for the optimizer. We propose a simple technique to speed up the training optimization by clustering features dynamically, and interleaving the standard optimizer with another, coarse-grained, faster optimizer that uses far fewer variables. Experiments with logistic regression training for text classification and conditional random field (CRF) training for information extraction show promising speed-ups between 2× and 9× without any systematic or significant degradation in the quality of the estimated models.

• Type: Conference
• Year: 2006
• Where: ICDM
• Authors: Arpit Mathur, Soumen Chakrabarti
• Keywords: Data Mining, Faster Optimizer, ICDM 2006, Logistic Regression Training, Robust Conditional Distributions