Multi-Reference Generative Face Video Compression with Contrastive Learning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Multi-Reference Generative Face Video Compression with Contrastive Learning

Goluck Konuko
Giuseppe Valenzise

Résumé

Generative face video coding (GFVC) has been demonstrated as a potential approach to low-latency, low bitrate video conferencing. GFVC frameworks achieve an extreme gain in coding efficiency with over 70% bitrate savings when compared to conventional codecs at bitrates below 10kbps. In recent MPEG/JVET standardization efforts, all the information required to reconstruct video sequences using GFVC frameworks are adopted as part of the supplemental enhancement information (SEI) in existing compression pipelines. In light of this development, we aim to address a challenge that has been weakly addressed in prior GFVC frameworks, i.e., reconstruction drift as the distance between the reference and target frames increases. This challenge creates the need to update the reference buffer more frequently by transmitting more Intrarefresh frames, which are the most expensive element of the GFVC bitstream. To overcome this problem, we propose instead multiple reference animation as a robust approach to minimizing reconstruction drift, especially when used in a bi-directional prediction mode. Further, we propose a contrastive learning formulation for multi-reference animation. We observe that using a contrastive learning framework enhances the representation capabilities of the animation generator. The resulting framework, MRDAC (Multi-Reference Deep Animation Codec) can therefore be used to compress longer sequences with fewer reference frames or achieve a significant gain in reconstruction accuracy at comparable bitrates to previous frameworks. Quantitative and qualitative results show significant coding and reconstruction quality gains compared to previous GFVC methods, and more accurate animation quality in presence of large pose and facial expression changes.

Fichier principal
Vignette du fichier
Multi_Reference_DAC_MMSP.pdf (3.25 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04683529 , version 1 (02-09-2024)

Licence

Identifiants

  • HAL Id : hal-04683529 , version 1

Citer

Goluck Konuko, Giuseppe Valenzise. Multi-Reference Generative Face Video Compression with Contrastive Learning. MMSP, IEEE Signal Processing Society, Oct 2024, WEST LAFAYETTE - INDIANA, United States. ⟨hal-04683529⟩
0 Consultations
0 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More