Supervising the Transfer of Reasoning Patterns in VQA - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Supervising the Transfer of Reasoning Patterns in VQA

Résumé

Methods for Visual Question Anwering (VQA) are notorious for leveraging dataset biases rather than performing reasoning, hindering generalization. It has been recently shown that better reasoning patterns emerge in attention layers of a state-of-the-art VQA model when they are trained on perfect (oracle) visual inputs. This provides evidence that deep neural networks can learn to reason when training conditions are favorable enough. However, transferring this learned knowledge to deployable models is a challenge, as much of it is lost during the transfer. We propose a method for knowledge transfer based on a regularization term in our loss function, supervising the sequence of required reasoning operations. We provide a theoretical analysis based on PAC-learning, showing that such program prediction can lead to decreased sample complexity under mild hypotheses. We also demonstrate the effectiveness of this approach experimentally on the GQA dataset and show its complementarity to BERT-like self-supervised pre-training.

Dates et versions

hal-04090622 , version 1 (05-05-2023)

Identifiants

Citer

Corentin Kervadec, Christian Wolf, Grigory Antipov, Moez Baccouche, Madiha Nadri. Supervising the Transfer of Reasoning Patterns in VQA. NeurIPS, Dec 2021, Virtual, France. ⟨hal-04090622⟩
10 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More