Communication Dans Un Congrès Année : 2018

Finding beans in burgers: Deep semantic-visual embedding with localization

Résumé

Several works have proposed to learn a two-path neural network that maps images and texts, respectively, to a same shared Euclidean space where geometry captures useful semantic relationships. Such a multi-modal embedding can be trained and used for various tasks, notably image captioning. In the present work, we introduce a new architecture of this type, with a visual path that leverages recent space-aware pooling mechanisms. Combined with a textual path which is jointly trained from scratch, our semantic-visual embedding offers a versatile model. Once trained under the supervision of captioned images, it yields new state-of-the-art performance on cross-modal retrieval. It also allows the localization of new concepts from the embedding space into any input image, delivering state-of-the-art result on the visual grounding of phrases.

Fichier principal
Vignette du fichier
findingbeansinburger.pdf (2.64 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence
Loading...

Dates et versions

hal-02171857 , version 1 (03-07-2019)

Licence

Identifiants

Citer

Martin Engilberge, Louis Chevallier, Patrick Pérez, Matthieu Cord. Finding beans in burgers: Deep semantic-visual embedding with localization. CVPR 2018 - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018, Salt Lake City, United States. pp.3984-3993, ⟨10.1109/CVPR.2018.00419⟩. ⟨hal-02171857⟩
184 Consultations
156 Téléchargements

Altmetric

Partager

  • More