Cold Posteriors through PAC-Bayes - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

Cold Posteriors through PAC-Bayes

Résumé

We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds. We argue that in the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance on out-of-sample data. Instead, out-of-sample error is better described through a generalization bound. In this context, we explore the connections of the ELBO objective from variational inference and the PAC-Bayes objectives. We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain a temperature parameter λ which is not restricted to be λ = 1. For both regression and classification tasks, in the case of isotropic Laplace approximations to the posterior, we show how this PAC-Bayesian interpretation of the temperature parameter captures important aspects of the cold posterior effect.
Fichier principal
Vignette du fichier
paper.pdf (892.02 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03791457 , version 1 (29-09-2022)

Identifiants

  • HAL Id : hal-03791457 , version 1

Citer

Konstantinos Pitas, Julyan Arbel. Cold Posteriors through PAC-Bayes. 2022. ⟨hal-03791457⟩
56 Consultations
57 Téléchargements

Partager

Gmail Facebook X LinkedIn More