Department of Scientific Computing, Florida State University
"Numerical Daemons of Monte Carlo Estimators of Bayesian Model Evidence"

Abstract:

We are inevitably dealing with conceptual and parametric uncertainties when modeling natural systems due to their heterogeneity and complexity. Which model to select? Although this simple question is elusive, Bayesian model evidence can provide an answer. By balancing the trade-off between the model fit and model complexity, Bayesian model evidence (BME) is the overall probability of the model in reproducing the observation data given all the parameter values that the model can assume. Thus given its simple definition, BME is a central tool in Bayesian multimodel analysis. To estimate BME, the Monte Carlo (MC) based numerical estimators are a promising choice, if their computational cost can be afforded. Theoretically, BME numerical estimators are bias-free as they hold no assumptions about the shape of the underlying distributions, yet their numerical implementation can introduce bias. This talk will discuss some bias sources of BME numerical estimators. Specifically, the talk focuses on the impact of rounding error on BME estimation, a topic that is often overlooked in literature. I will present two sources of errors, namely acceptance-ratio rounding error that impacts the Metropolis acceptance ratio during the Markov chain Monte Carol sampling (MCMC) and likelihood rounding error that occurs when evaluating the likelihood function. These two daemons arise from arithmetic precision of floating point numbers. This talk illustrates the conditions that invoke these daemons and their implications on BME estimation. I will show how likelihood rounding error can result in severe underestimation of BME, while acceptance-ratio rounding error can result in overestimation of BME. I test the rounding error vulnerability of two popular numerical estimators, namely, arithmetic mean (AM) and harmonic mean (HM), and two state of the art numerical estimators, namely, thermodynamic integration (TI) and steppingstone sampling (SS). Results indicate that HM and SS are less vulnerable to rounding error, yet HM is a biased estimator. AM and TI turned out to be vulnerable, resulting in underestimation and overestimation, respectively. The discussion will provide insights on how to check BME estimates for errors to reduce frustration when getting contradicting BME estimates from different numerical estimators.