Conditional cross correlation network for video question answering - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Conditional cross correlation network for video question answering

Résumé

Video question answering (VideoQA) is the process that aims at responding to questions expressed in natural language, according to the semantic content of a given video. VideoQA is a highly challenging task and demands a comprehensive understanding of the video document, including the recognition of the various objects, actions and activities involved together with the spatial, temporal and causal relations between them. To tackle the challenge of VideoQA, most methods propose efficient techniques to fuse the representations between visual and textual modalities. In this paper, we introduce a novel framework based on a conditional cross-correlation network that learns multimodal contextualization with reduced computational and memory requirements. At the core of our approach, we consider a cross-correlation module designed to learn reciprocally constrained visual/textual features combined with a lightweight transformer that fuses the intermodal contextualization between visual and textual modalities. We test the vulnerability of the composing elements of our pipeline using black box attacks. To this purpose, we automatically generate semantic-preserving rephrased questions. The ablation study conducted confirms the importance of each module in the framework. The experimental evaluation, carried out on the MSVD-QA benchmark, validates the proposed methodology with average accuracy scores of 43.58%. When compared with state-of-the-art methods the proposed method yields gains in accuracy of more than 4%and achieves a 43.58% accuracy rate on the MSVD-QA data set.
Fichier principal
Vignette du fichier
Conditional Cross Correlation Network for Video Question Answering.pdf (1.06 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04305444 , version 1 (14-02-2024)

Identifiants

Citer

Kaouther Ouenniche, Ruxandra Tapu, Titus Zaharia. Conditional cross correlation network for video question answering. 2023 IEEE 17th International Conference on Semantic Computing (ICSC), Feb 2023, Laguna Hills, United States. pp.25-32, ⟨10.1109/ICSC56153.2023.00011⟩. ⟨hal-04305444⟩
7 Consultations
3 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More