Gradient scarcity with Bilevel Optimization for Graph Learning - Institut de Mathématiques de Bourgogne Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

Gradient scarcity with Bilevel Optimization for Graph Learning

Résumé

A common issue in graph learning under the semi-supervised setting is referred to as gradient scarcity. That is, learning graphs by minimizing a loss on a subset of nodes causes edges between unlabelled nodes that are far from labelled ones to receive zero gradients. The phenomenon was first described when optimizing the graph and the weights of a Graph Neural Network (GCN) with a joint optimization algorithm. In this work, we give a precise mathematical characterization of this phenomenon, and prove that it also emerges in bilevel optimization, where additional dependency exists between the parameters of the problem. While for GCNs gradient scarcity occurs due to their finite receptive field, we show that it also occurs with the Laplacian regularization model, in the sense that gradients amplitude decreases exponentially with distance to labelled nodes. To alleviate this issue, we study several solutions: we propose to resort to latent graph learning using a Graph-to-Graph model (G2G), graph regularization to impose a prior structure on the graph, or optimizing on a larger graph than the original one with a reduced diameter. Our experiments on synthetic and real datasets validate our analysis and prove the efficiency of the proposed solutions.
Fichier principal
Vignette du fichier
gradient_scarcity.pdf (1.53 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04041721 , version 1 (22-03-2023)

Identifiants

Citer

Hashem Ghanem, Samuel Vaiter, Nicolas Keriven. Gradient scarcity with Bilevel Optimization for Graph Learning. 2023. ⟨hal-04041721⟩
18 Consultations
21 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More