Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference - Archive ouverte HAL
Pré-Publication, Document De Travail Année : 2023

Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference

Loi des grands nombres pour un réseau de neurones bayésien à deux couches entraîné par inférence variationnelle

Arnaud Descours
  • Fonction : Auteur
  • PersonId : 1151874
Tom Huix
  • Fonction : Auteur
  • PersonId : 1268148
Manon Michel
Éric Moulines
Boris Nectoux
  • Fonction : Auteur
  • PersonId : 1151875

Résumé

We provide a rigorous analysis of training by variational inference (VI) of Bayesian neural networks in the two-layer and infinite-width case. We consider a regression problem with a regularized evidence lower bound (ELBO) which is decomposed into the expected log-likelihood of the data and the Kullback-Leibler (KL) divergence between the a priori distribution and the variational posterior. With an appropriate weighting of the KL, we prove a law of large numbers for three different training schemes: (i) the idealized case with exact estimation of a multiple Gaussian integral from the reparametrization trick, (ii) a minibatch scheme using Monte Carlo sampling, commonly known as Bayes by Backprop, and (iii) a new and computationally cheaper algorithm which we introduce as Minimal VI. An important result is that all methods converge to the same mean-field limit. Finally, we illustrate our results numerically and discuss the need for the derivation of a central limit theorem.
Fichier principal
Vignette du fichier
elbo_camera_v1.pdf (539.44 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04153801 , version 1 (06-07-2023)

Identifiants

Citer

Arnaud Descours, Tom Huix, Arnaud Guillin, Manon Michel, Éric Moulines, et al.. Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference. 2023. ⟨hal-04153801⟩
54 Consultations
72 Téléchargements

Altmetric

Partager

More