On the convergence of a stochastic approximation method for structured bi-level optimization - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2018

On the convergence of a stochastic approximation method for structured bi-level optimization

Résumé

We analyze the convergence of stochastic gradient methods for well structured bi-level optimization problems. We address two specific cases: first when the outer objective function can be expressed as a finite sum of independent terms, and next when both the outer and inner objective functions can be expressed as finite sums of independent terms. We assume Lipschitz continuity and differentiability of both objectives as well as convexity of the inner objective and consider diminishing steps sizes. We show that, under these conditions and some other assumptions on the implicit function and the variance of the gradient errors, both methods converge in expectation to a stationary point of the problem if gradient approximations are chosen so as to satisfy a sufficient decrease condition. We also discuss the satisfaction of our assumptions in machine learning problems where these methods can be nicely applied to automatically tune hyperparameters when the loss functions are very large sums of error terms.
Fichier principal
Vignette du fichier
Bilevel_preprint.pdf (317.15 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-01932372 , version 1 (23-11-2018)

Identifiants

  • HAL Id : hal-01932372 , version 1

Citer

Nicolas Couellan, Wenjuan Wang. On the convergence of a stochastic approximation method for structured bi-level optimization. 2018. ⟨hal-01932372⟩
163 Consultations
296 Téléchargements

Partager

Gmail Facebook X LinkedIn More