Performance Guarantees for Regularized Maximum Entropy Density Estimation

Abstract. We consider the problem of estimating an unknown probability distribution from samples using the principle of maximum entropy (maxent). To alleviate overfitting with a very large number of features, we propose applying the maxent principle with relaxed constraints on the expectations of the features. By convex duality, this turns out to be equivalent to finding the Gibbs distribution minimizing a regularized version of the empirical log loss. We prove nonasymptotic bounds showing that, with respect to the true underlying distribution, this relaxed version of maxent produces density estimates that are almost as good as the best possible. These bounds are in terms of the deviation of the feature empirical averages relative to their true expectations, a number that can be bounded using standard uniform-convergence techniques. In particular, this leads to bounds that drop quickly with the number of samples, and that depend very moderately on the number or complexity of the feat...

• Type: Conference
• Year: 2004
• Where: COLT
• Authors: Miroslav Dudík, Steven J. Phillips, Robert E. Schapire
• Keywords: COLT 2004, Maxent Principle, Maxent Produces Density, Unknown Probability Distribution