SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation

Résumé

Our goal is to synthesize 3D human motions given textual inputs describing simultaneous actions, for example 'waving hand' while 'walking' at the same time. We refer to generating such simultaneous movements as performing 'spatial compositions'. In contrast to temporal compositions that seek to transition from one action to another, spatial compositing requires understanding which body parts are involved in which action, to be able to move them simultaneously. Motivated by the observation that the correspondence between actions and body parts is encoded in powerful language models, we extract this knowledge by prompting GPT-3 with text such as "what are the body parts involved in the action ?", while also providing the parts list and few-shot examples. Given this action-part mapping, we combine body parts from two motions together and establish the first automated method to spatially compose two actions. However, training data with compositional actions is always limited by the combinatorics. Hence, we further create synthetic data with this approach, and use it to train a new state-of-the-art text-to-motion generation model, called SINC ("SImultaneous actioN Compositions for 3D human motions"). In our experiments, that training with such GPT-guided synthetic data improves spatial composition generation over baselines. Our code is publicly available at https://sinc.is.tue.mpg.de/.
Fichier principal
Vignette du fichier
2304.10417.pdf (4.24 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04258194 , version 1 (25-10-2023)

Identifiants

Citer

Nikos Athanasiou, Mathis Petrovich, Michael J. Black, Gül Varol. SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation. International Conference on Computer Vision, Oct 2023, Paris, France. ⟨hal-04258194⟩
166 Consultations
14 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More