ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities

Résumé

Multimodality has recently gained attention in the medical domain, where imaging or video modalities may be integrated with biomedical signals or health records. Yet, two challenges remain: balancing the contributions of modalities, especially in cases with a limited amount of data available, and tackling missing modalities. To address both issues, in this paper, we introduce the AnchoreD multimodAl Physiological Transformer (ADAPT), a multimodal, scalable framework with two key components: (i) aligning all modalities in the space of the strongest, richest modality (called anchor) to learn a joint embedding space, and (ii) a Masked Multimodal Transformer, leveraging both inter- and intra-modality correlations while handling missing modalities. We focus on detecting physiological changes in two real-life scenarios: stress in individuals induced by specific triggers and fighter pilots' loss of consciousness induced by g-forces. We validate the generalizability of ADAPT through extensive experiments on two datasets for these tasks, where we set the new state of the art while demonstrating its robustness across various modality scenarios and its high potential for real-life applications. Our code is available at https://github.com/jumdc/ADAPT.git.
Fichier principal
Vignette du fichier
MIDL (1).pdf (1.08 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04597429 , version 1 (02-06-2024)

Licence

Identifiants

  • HAL Id : hal-04597429 , version 1

Citer

Julie Mordacq, Leo Milecki, Maria Vakalopoulou, Steve Oudot, Vicky Kalogeiton. ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities. MIDL 2024 - Medical Imaging with Deep Learning, Jul 2024, Paris, France. ⟨hal-04597429⟩
15 Consultations
8 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More