Gradient Masked Averaging for Federated Learning - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Transactions on Machine Learning Research Journal Année : 2022

Gradient Masked Averaging for Federated Learning

Irene Tenison
  • Fonction : Auteur
Sai Aravind Sreeramadas
  • Fonction : Auteur
Vaikkunth Mugunthan
  • Fonction : Auteur
Edouard Oyallon
Eugene Belilovsky
  • Fonction : Auteur
Irina Rish
  • Fonction : Auteur

Résumé

Federated learning is an emerging paradigm that permits a large number of clients with heterogeneous data to coordinate learning of a unified global model without the need to share data amongst each other. Standard federated learning algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. However, in heterogeneous settings averaging can result in information loss and lead to poor generalization due to the bias induced by dominant clients. We hypothesize that to generalize better across non-i.i.d datasets as in FL settings, the algorithms should focus on learning the invariant mechanism that is constant while ignoring spurious mechanisms that differ across clients. Inspired from recent work in the Out-of-Distribution literature, we propose a gradient masked averaging approach for federated learning as an alternative to the standard averaging of client updates. This client update aggregation technique can be adapted as a drop-in replacement in most existing federated algorithms. We perform extensive experiments with gradient masked approach on multiple FL algorithms with in-distribution, real-world, and out-of-distribution (as the worst case scenario) test dataset and show that it provides consistent improvements, particularly in the case of heterogeneous clients.

Dates et versions

hal-04262447 , version 1 (27-10-2023)

Identifiants

Citer

Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Eugene Belilovsky, et al.. Gradient Masked Averaging for Federated Learning. Transactions on Machine Learning Research Journal, 2022. ⟨hal-04262447⟩
12 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More