Going Beyond Nouns With Vision & Language Models Using Synthetic Data - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Going Beyond Nouns With Vision & Language Models Using Synthetic Data

Résumé

Large-scale pre-trained Vision & Language (VL) models have shown remarkable performance in many applications, enabling replacing a fixed set of supported classes with zero-shot open vocabulary reasoning over (almost arbitrary) natural language prompts. However, recent works have uncovered a fundamental weakness of these models. For example, their difficulty to understand Visual Language Concepts (VLC) that go 'beyond nouns' such as the meaning of non-object words (e.g., attributes, actions, relations, states, etc.), or difficulty in performing compositional reasoning such as understanding the significance of the order of the words in a sentence. In this work, we investigate to which extent purely synthetic data could be leveraged to teach these models to overcome such shortcomings without compromising their zero-shot capabilities. We contribute Synthetic Visual Concepts (SyViC) - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models. Additionally, we propose a general VL finetuning strategy for effectively leveraging SyViC towards achieving these improvements. Our extensive experiments and ablations on VL-Checklist, Winoground, and ARO benchmarks demonstrate that it is possible to adapt strong pre-trained VL models with synthetic data significantly enhancing their VLC understanding (e.g. by 9.9% on ARO and 4.3% on VL-Checklist) with under 1% drop in their zero-shot accuracy.

Dates et versions

hal-04323567 , version 1 (05-12-2023)

Identifiants

Citer

Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, et al.. Going Beyond Nouns With Vision & Language Models Using Synthetic Data. International Conference on Computer Vision (ICCV) 2023, Oct 2023, Paris, France. ⟨hal-04323567⟩
81 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More