Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks

Résumé

Vision-language foundation models such as CLIP have shown impressive zero-shot performance on many tasks and datasets, especially thanks to their free-text inputs. However, they struggle to handle some downstream tasks, such as fine-grained attribute detection and localization. In this paper, we propose a multitask fine-tuning strategy based on a positive/negative prompt formulation to further leverage the capacities of the vision-language foundation models. Using the CLIP architecture as baseline, we show strong improvements on bird fine-grained attribute detection and localization tasks, while also increasing the classification performance on the CUB200-2011 dataset. We provide source code for reproducibility purposes: it is available at https://github.com/FactoDeepLearning/MultitaskVLFM.

Dates et versions

hal-04162923 , version 1 (17-07-2023)

Identifiants

Citer

Denis Coquenet, Clément Rambour, Emanuele Dalsasso, Nicolas Thome. Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks. 2023. ⟨hal-04162923⟩
22 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More