Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis

Résumé

We propose a new multimodal variational autoencoder that enables to generate from the joint distribution and conditionally to any number of complex modalities. The unimodal posteriors are conditioned on the Deep Canonical Correlation Analysis embeddings which preserve the shared information across modalities leading to more coherent cross-modal generations. Furthermore, we use Normalizing Flows to enrich the unimodal posteriors and achieve more diverse data generation. Finally, we propose to use a Product of Experts for inferring one modality from several others which makes the model scalable to any number of modalities. We demonstrate that our method improves likelihood estimates, diversity of the generations and in particular coherence metrics in the conditional generations on several datasets.
Fichier principal
Vignette du fichier
icml2022-3.pdf (14.8 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04207134 , version 1 (14-09-2023)

Identifiants

Citer

Agathe Senellart, Clément Chadebec, Stéphanie Allassonnière, Stéphanie Allassonnière. Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis. 2023. ⟨hal-04207134⟩
21 Consultations
9 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More