Calibrating Probability Density Forecasts with Multi-Objective Search

Abstract. In this paper, we show that the optimization of density forecasting models for regression in machine learning can be formulated as a multi-objective problem. We describe the two objectives of sharpness and calibration and suggest suitable scoring metrics for both. We use the popular negative log-likelihood as a measure of sharpness and the probability integral transform as a measure of calibration. We show how optimization on negative log-likelihood alone often results in sub-optimal models. To solve this problem we introduce a multi-objective evolutionary optimization framework that can produce better density forecasts from a prediction users perspective. Our experiments show improvements over state-of-the-art approaches.

• Type: Conference
• Year: 2006
• Where: ECAI
• Authors: Michael Carney, Padraig Cunningham
• Keywords: Artificial Intelligence, Density Forecasting Models, ECAI 2006, Negative Log-likelihood, Suitable Scoring Metrics