Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation

Résumé

Stochastic Gradient Descent (SGD) with adaptive steps is now widely used for training deep neural networks. Most theoretical results assume access to unbiased gradient estimators, which is not the case in several recent deep learning and reinforcement learning applications that use Monte Carlo methods. This paper provides a comprehensive non-asymptotic analysis of SGD with biased gradients and adaptive steps for convex and non-convex smooth functions. Our study incorporates time-dependent bias and emphasizes the importance of controlling the bias and Mean Squared Error (MSE) of the gradient estimator. In particular, we establish that Adagrad and RMSProp with biased gradients converge to critical points for smooth non-convex functions at a rate similar to existing results in the literature for the unbiased case. Finally, we provide experimental results using Variational Autoenconders (VAE) that illustrate our convergence results and show how the effect of bias can be reduced by appropriate hyperparameter tuning.
Fichier principal
Vignette du fichier
BiasedSGD.pdf (1.42 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04435027 , version 1 (02-02-2024)

Identifiants

Citer

Sobihan Surendran, Antoine Godichon-Baggioni, Adeline Fermanian, Sylvain Le Corff. Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation. 2024. ⟨hal-04435027v1⟩
13 Consultations
6 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More