On the Margin Explanation of Boosting Algorithms

Much attention has been paid to the theoretical explanation of the empirical success of AdaBoost. The most influential work is the margin theory, which is essentially an upper bound for the generalization error of any voting classifier in terms of the margin distribution over the training data. However, Breiman raised important questions about the margin explanation by developing a boosting algorithm arc-gv that provably generates a larger minimum margin than AdaBoost. He also gave a sharper bound in terms of the minimum margin, and argued that the minimum margin governs the generalization. In experiments however, arc-gv usually performs worse than AdaBoost, putting the margin explanation into serious doubts. In this paper, we try to give a complete answer to Breiman's critique by proving a bound in terms of a new margin measure called Equilibrium margin (Emargin). The Emargin bound is uniformly sharper than Breiman's minimum margin bound. This result suggests that the minim...

• Type: Conference
• Year: 2008
• Where: COLT
• Authors: Liwei Wang, Masashi Sugiyama, Cheng Yang, Zhi-Hua Zhou, Jufu Feng
• Keywords: COLT 2008, Larger Minimum Margin, Machine Learning, Margin Explanation, Minimum Margin