TS-NET: COMBINING MODALITY SPECIFIC AND COMMON FEATURES FOR MULTIMODAL PATCH MATCHING - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2018

TS-NET: COMBINING MODALITY SPECIFIC AND COMMON FEATURES FOR MULTIMODAL PATCH MATCHING

Résumé

Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra constraints in the intermediate layers of such networks further boosts the performance. Experimentations on three multimodal datasets show significant performance gains in comparison with Siamese and Pseudo-Siamese networks.
Fichier principal
Vignette du fichier
main.pdf (220 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01804551 , version 1 (31-05-2018)

Identifiants

Citer

Sovann En, Alexis Lechervy, Frédéric Jurie. TS-NET: COMBINING MODALITY SPECIFIC AND COMMON FEATURES FOR MULTIMODAL PATCH MATCHING. ICIP, IEEE International Conference on Image Processing, Oct 2018, Athens, Greece. ⟨hal-01804551⟩
217 Consultations
224 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More