Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering

Résumé

The Knowledge-Aware Visual Question Answering about Entity task aims to disambiguate entities using textual and visual information, as well as knowledge. It usually relies on two independent steps, information retrieval then reading comprehension, that do not benefit each other. Retrieval Augmented Generation (RAG) offers a solution by using generated answers as feedback for retrieval training. RAG usually relies solely on pseudo-relevant passages retrieved from external knowledge bases which can lead to ineffective answer generation. In this work, we propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion. We formulate a joint-training RAG loss such that answer generation is conditioned on both entity and passage retrievals. We show through experiments new state-of-the-art performance on the VIQuAE KB-VQA benchmark and demonstrate that our approach can help retrieve more actual relevant knowledge to generate accurate answers.
Fichier principal
Vignette du fichier
2024.emnlp-main.922.pdf (3.07 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-04852275 , version 1 (20-12-2024)

Licence

Identifiants

Citer

Omar Adjali, Olivier Ferret, Sahar Ghannay, Hervé Le Borgne. Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024, Miami, United States. pp.16499-16513, ⟨10.18653/v1/2024.emnlp-main.922⟩. ⟨hal-04852275⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More