eP-ALM: Efficient Perceptual Augmentation of Language Models - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

eP-ALM: Efficient Perceptual Augmentation of Language Models

Résumé

Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code will be available here: https://github.com/mshukor/eP-ALM.
Fichier principal
Vignette du fichier
2303.11403.pdf (2.04 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04232603 , version 1 (08-10-2023)

Identifiants

Citer

Mustafa Shukor, Corentin Dancette, Matthieu Cord. eP-ALM: Efficient Perceptual Augmentation of Language Models. International Conference on Computer Vision (ICCV23), Oct 2023, Paris, France. pp.22056-22069, ⟨10.48550/arXiv.2303.11403⟩. ⟨hal-04232603⟩
13 Consultations
39 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More