Improving generalization in large language models by learning prefix subspaces - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Improving generalization in large language models by learning prefix subspaces

Résumé

This article focuses on large language models (LLMs) fine-tuning in the scarce data regime (also known as the "few-shot" learning setting). We propose a method to increase the generalization capabilities of LLMs based on neural network subspaces. This optimization method, recently introduced in computer vision, aims to improve model generalization by identifying wider local optima through the joint optimization of an entire simplex of models in parameter space. Its adaptation to massive, pretrained transformers, however, poses some challenges. First, their considerable number of parameters makes it difficult to train several models jointly, and second, their deterministic parameter initialization schemes make them unfit for the subspace method as originally proposed. We show in this paper that "Parameter Efficient Fine-Tuning" (PEFT) methods, however, are perfectly compatible with this original approach, and propose to learn entire simplex of continuous prefixes. We test our method on a variant of the GLUE benchmark adapted to the few-shot learning setting, and show that both our contributions jointly lead to a gain in average performances compared to sota methods. The implementation can be found at the following link: https: //github.com/Liloulou/prefix_subspace
Fichier principal
Vignette du fichier
2310.15793.pdf (265.86 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04286331 , version 1 (15-11-2023)

Identifiants

  • HAL Id : hal-04286331 , version 1

Citer

Louis Falissard, Vincent Guigue, Laure Soulier. Improving generalization in large language models by learning prefix subspaces. 61st Annual Meeting of the Association for Computational Linguistics (2023), Association for Computational Linguistics, Jul 2023, Toronto, Canada. ⟨hal-04286331⟩
63 Consultations
26 Téléchargements

Partager

Gmail Facebook X LinkedIn More