A Lower Bound for Agnostically Learning Disjunctions

We prove that the concept class of disjunctions cannot be pointwise approximated by linear combinations of any small set of arbitrary real-valued functions. That is, suppose there exist functions φ1,...,φr : {−1,1}n → R with the property that every disjunction f on n variables has f −∑r i=1 αiφi ∞ 1/3 for some reals α1,...,αr. We prove that then r 2Ω( √ n). This lower bound is tight. We prove an incomparable lower bound for the concept class of linear-size DNF formulas. For the concept class of majority functions, we obtain a lower bound of Ω(2n/n), which almost meets the trivial upper bound of 2n for any concept class. These lower bounds substantially strengthen and generalize the polynomial approximation lower bounds of Paturi and show that the regression-based agnostic learning algorithm of Kalai et al. is optimal. Our techniques involve a careful application of results in communication complexity due to Razborov and Buhrman et al.

• Type: Conference
• Year: 2007
• Where: COLT
• Authors: Adam R. Klivans, Alexander A. Sherstov
• Keywords: COLT 2007, Concept Class, Incomparable Lower Bound, Lower Bounds, Machine Learning