Can Entropic Regularization Be Replaced by Squared Euclidean Distance Plus Additional Linear Constraints

There are two main families of on-line algorithms depending on whether a relative entropy or a squared Euclidean distance is used as a regularizer. The difference between the two families can be dramatic. The question is whether one can always achieve comparable performance by replacing the relative entropy regularization by the squared Euclidean distance plus additional linear constraints. We formulate a simple open problem along these lines for the case of learning disjunctions. Assume the target concept is a k literal disjunction over n variables. The instances are bit vectors x {0, 1}n and the disjunction Vi1 Vi2 . . . Vik is true on instance x iff at least one bit in the positions i1, i2, . . . , ik is one. We can represent the above disjunction as a weight vector w: all relevant weights wij are set to some threshold > 0 and the remaining n - k irrelevant weights are zero. Now the disjunction is a linear threshold function: the disjunction is true on x iff w

• Type: Conference
• Year: 2006
• Where: COLT
• Authors: Manfred K. Warmuth
• Keywords: COLT 2006, Disjunction, Machine Learning, Relative Entropy, Squared Euclidean Distance