Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities

In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate, as it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be used to compare models, for example, by computing the probability of one model having a better expected utility than some other model. We propose an approach using crossvalidation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions. We also discuss the probabilistic assumptions made and properties of two practical cross-validation methods, importance sampling and k-fold cross-validation. As illustrative examples, we use MLP neural...

• Type: Journal
• Year: 2002
• Where: NECO
• Authors: Aki Vehtari, Jouko Lampinen
• Keywords: Better Expected Utility, Expected Utility, NECO 2002, Utility Estimates