Beyond spectral gap (extended): The role of the topology in decentralized learning - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

Beyond spectral gap (extended): The role of the topology in decentralized learning

Résumé

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers communicate over a sparse graph, current theory fails to capture important aspects of real-world behavior. First, the `spectral gap' of the communication graph is not predictive of its empirical performance in (deep) learning. Second, current theory does not explain that collaboration enables larger learning rates than training alone. In fact, it prescribes smaller learning rates, which further decrease as graphs become larger, failing to explain convergence dynamics in infinite graphs. This paper aims to paint an accurate picture of sparsely-connected distributed optimization. We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives. Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies. This paper is an extension of the conference paper by Vogels et. al. (2022). Code: https://github.com/epfml/topology-in-decentralized-learning.
Fichier principal
Vignette du fichier
2301.02151.pdf (988.03 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04107280 , version 1 (26-05-2023)

Identifiants

Citer

Thijs Vogels, Hadrien Hendrikx, Martin Jaggi. Beyond spectral gap (extended): The role of the topology in decentralized learning. 2023. ⟨hal-04107280⟩
75 Consultations
58 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More