On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions

Résumé

We investigate the out-of-domain generalization of random feature (RF) models and Transformers. We first prove that in the `generalization on the unseen (GOTU)' setting, where training data is fully seen in some part of the domain but testing is made on another part, and for RF models in the small feature regime, the convergence takes place to interpolators of minimal degree as in the Boolean case (Abbe et al., 2023). We then consider the sparse target regime and explain how this regime relates to the small feature regime, but with a different regularization term that can alter the picture in the non-Boolean case. We show two different outcomes for the sparse regime with q-ary data tokens: (1) if the data is embedded with roots of unities, then a min-degree interpolator is learned like in the Boolean case for RF models, (2) if the data is not embedded as such, e.g., simply as integers, then RF models and Transformers may not learn minimal degree interpolators. This shows that the Boolean setting and its roots of unities generalization are special cases where the minimal degree interpolator offers a rare characterization of how learning takes place. For more general integer and real-valued settings, a more nuanced picture remains to be fully characterized.
Fichier principal
Vignette du fichier
ICML_2024_camera_ready-1.pdf (653.04 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04619375 , version 1 (20-06-2024)

Identifiants

  • HAL Id : hal-04619375 , version 1

Citer

Denys Pushkin, Raphaël Berthier, Emmanuel Abbe. On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions. 2024. ⟨hal-04619375⟩
0 Consultations
0 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More