Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization

Résumé

In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to as a "mixture". Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression "mixture", and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as visual-linguistic ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of visual-linguistic ambiguity, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization as compared to a strong single-task baseline.
Fichier principal
Vignette du fichier
emnlp23impress.pdf (4.64 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04259861 , version 1 (26-10-2023)

Identifiants

  • HAL Id : hal-04259861 , version 1

Citer

Cennet Oguz, Pascal Denis, Emmanuel Vincent, Simon Ostermann, Josef van Genabith. Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization. 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023, Singapore, Singapore. ⟨hal-04259861⟩
103 Consultations
55 Téléchargements

Partager

Gmail Facebook X LinkedIn More