Mistake Bounds for Maximum Entropy Discrimination

We establish a mistake bound for an ensemble method for classification based on maximizing the entropy of voting weights subject to margin constraints. The bound is the same as a general bound proved for the Weighted Majority Algorithm, and similar to bounds for other variants of Winnow. We prove a more refined bound that leads to a nearly optimal algorithm for learning disjunctions, again, based on the maximum entropy principle. We describe a simplification of the on-line maximum entropy method in which, after each iteration, the margin constraints are replaced with a single linear inequality. The simplified algorithm, which takes a similar form to Winnow, achieves the same mistake bounds.

• Type: Conference
• Year: 2004
• Where: NIPS
• Authors: Philip M. Long, Xinyu Wu
• Keywords: Margin Constraints, Maximum Entropy, Mistake Bounds, NIPS 2004, NIPS 2007