Pré-Publication, Document De Travail Année : 2025

Diagonal Linear Networks and the Lasso Regularization Path

Résumé

Diagonal linear networks are neural networks with linear activation and diagonal weight matrices. Their theoretical interest is that their implicit regularization can be rigorously analyzed: from a small initialization, the training of diagonal linear networks converges to the linear predictor with minimal 1-norm among minimizers of the training loss. In this paper, we deepen this analysis showing that the full training trajectory of diagonal linear networks is closely related to the lasso regularization path. In this connection, the training time plays the role of an inverse regularization parameter. Both rigorous results and simulations are provided to illustrate this conclusion. Under a monotonicity assumption on the lasso regularization path, the connection is exact while in the general case, we show an approximate connection.

Fichier principal
Vignette du fichier
DLN_Lasso.pdf (841.16 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05282524 , version 1 (25-09-2025)
hal-05282524 , version 2 (28-01-2026)
hal-05282524 , version 3 (18-03-2026)

Licence

Identifiants

  • HAL Id : hal-05282524 , version 3

Citer

Raphaël Berthier. Diagonal Linear Networks and the Lasso Regularization Path. 2026. ⟨hal-05282524v3⟩
1067 Consultations
384 Téléchargements

Partager

  • More