Supervised contrastive learning as multi-objective optimization for fine-tuning large pre-trained language models - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Supervised contrastive learning as multi-objective optimization for fine-tuning large pre-trained language models

Résumé

Recently, Supervised Contrastive Learning (SCL) has been shown to significantly outperform the well-known crossentropy loss-based learning on most classification tasks. In SCL, a neural network is trained to optimize two objectives: pull an anchor and positive samples together in the embedding space, and push the anchor apart from the negatives. These two different objectives may be conflicting with one another, thus requiring a trade-off between them during optimization. In this work, we formulate the SCL problem as a Multi-Objective Optimization problem for the fine-tuning phase of RoBERTa language model. Two methods are utilized to solve the optimization problem: (i) the linear scalarization (LS) method, which minimizes a weighted linear combination of per-task losses; and (ii) the Exact Pareto Optimal (EPO) method which finds the intersection of the Pareto front with a given preference vector. We evaluate our approach on several GLUE benchmark tasks, without using data augmentations, memory banks, or generating adversarial examples. The empirical results show that the proposed learning strategy significantly outperforms a strong competitive contrastive learning baseline.
Fichier principal
Vignette du fichier
icassp2023.pdf (569.06 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04000223 , version 1 (22-02-2023)

Identifiants

  • HAL Id : hal-04000223 , version 1

Citer

Youness Moukafih, Mounir Ghogho, Kamel Smaïli. Supervised contrastive learning as multi-objective optimization for fine-tuning large pre-trained language models. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), Jun 2023, Rhodès, Greece. ⟨hal-04000223⟩
13 Consultations
75 Téléchargements

Partager

Gmail Facebook X LinkedIn More