Sparse Algorithms Are Not Stable: A No-Free-Lunch Theorem

Abstract—We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties contradict each other. That is, a sparse algorithm can not be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that ℓ1-regularized regression (Lasso) cannot be stable, while ℓ2-regularized regression is known to have strong stability properties and is therefore not sparse.

• Type: Journal
• Year: 2012
• Where: PAMI
• Authors: Huan Xu, Constantine Caramanis, Shie Mannor
• Keywords: Generalization Ability, PAMI 2012, Software Engineering, Sparsity, Stability Properties