Combining GAN with reverse correlation to construct personalized facial expressions - Archive ouverte HAL
Article Dans Une Revue PLoS ONE Année : 2023

Combining GAN with reverse correlation to construct personalized facial expressions

Résumé

Recent deep-learning techniques have made it possible to manipulate facial expressions in digital photographs or videos, however, these techniques still lack fine and personalized ways to control their creation. Moreover, current technologies are highly dependent on large labeled databases, which limits the range and complexity of expressions that can be modeled. Thus, these technologies cannot deal with non-basic emotions. In this paper, we propose a novel interdisciplinary approach combining the Generative Adversarial Network (GAN) with a technique inspired by cognitive sciences, psychophysical reverse correlation. Reverse correlation is a data-driven method able to extract an observer's 'mental representation' of what a given facial expression should look like. Our approach can generate 1) personalized facial expression prototypes, 2) of basic emotions, and non-basic emotions that are not available in existing databases, and 3) without the need for expertise. Personalized prototypes obtained with reverse correlation can then be applied to manipulate facial expressions. In addition, our system challenges the universality of facial expression prototypes by proposing the concepts of dominant and complementary action units to describe facial expression prototypes. The evaluations we conducted on a limited number of emotions validate the effectiveness of our proposed method. The code is available at https://github.com/yansen0508/Mental-Deep-Reverse-Engineering.
Fichier principal
Vignette du fichier
YAN_Sen_PLOS_One_preprint.pdf (5.12 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04183020 , version 1 (18-08-2023)

Identifiants

Citer

Sen Yan, Catherine Soladié, Jean-Julien Aucouturier, Renaud Seguier. Combining GAN with reverse correlation to construct personalized facial expressions. PLoS ONE, 2023, Affective Computing and Human-Computer Interactions, 18 (8), pp.e0290612. ⟨10.1371/journal.pone.0290612⟩. ⟨hal-04183020⟩
117 Consultations
78 Téléchargements

Altmetric

Partager

More