Distributed Learning with Sparse Communications by Identification - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue SIAM Journal on Mathematics of Data Science Année : 2021

Distributed Learning with Sparse Communications by Identification

Dmitry Grishchenko
  • Fonction : Auteur
  • PersonId : 1040116
Franck Iutzeler
Jérôme Malick

Résumé

In distributed optimization for large-scale learning, a major performance limitation comes from the communications between the different entities. When computations are performed by workers on local data while a coordinator machine coordinates their updates to minimize a global loss, we present an asynchronous optimization algorithm that efficiently reduces the communications between the coordinator and workers. This reduction comes from a random sparsification of the local updates. We show that this algorithm converges linearly in the strongly convex case and also identifies optimal strongly sparse solutions. We further exploit this identification to propose an automatic dimension reduction, aptly sparsifying all exchanges between coordinator and workers.
Fichier principal
Vignette du fichier
distSparse_arxiv.pdf (7.23 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-01950120 , version 1 (10-12-2018)
hal-01950120 , version 2 (10-01-2022)

Identifiants

Citer

Dmitry Grishchenko, Franck Iutzeler, Jérôme Malick, Massih-Reza Amini. Distributed Learning with Sparse Communications by Identification. SIAM Journal on Mathematics of Data Science, 2021, 3 (2), pp.715-735. ⟨10.1137/20M1347772⟩. ⟨hal-01950120v2⟩
224 Consultations
179 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More