On the relationship between multivariate splines and infinitely-wide neural networks - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

On the relationship between multivariate splines and infinitely-wide neural networks

Francis Bach
  • Fonction : Auteur
  • PersonId : 863086

Résumé

We consider multivariate splines and show that they have a random feature expansion as infinitely wide neural networks with one-hidden layer and a homogeneous activation function which is the power of the rectified linear unit. We show that the associated function space is a Sobolev space on a Euclidean ball, with an explicit bound on the norms of derivatives. This link provides a new random feature expansion for multivariate splines that allow efficient algorithms. This random feature expansion is numerically better behaved than usual random Fourier features, both in theory and practice. In particular, in dimension one, we compare the associated leverage scores to compare the two random expansions and show a better scaling for the neural network expansion.
Fichier principal
Vignette du fichier
neural_networks_splines.pdf (1.62 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03974669 , version 1 (06-02-2023)
hal-03974669 , version 2 (28-02-2023)

Identifiants

Citer

Francis Bach. On the relationship between multivariate splines and infinitely-wide neural networks. 2023. ⟨hal-03974669v2⟩
92 Consultations
247 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More