View-invariant Skeleton Action Representation Learning via Motion Retargeting - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue International Journal of Computer Vision Année : 2024

View-invariant Skeleton Action Representation Learning via Motion Retargeting

Résumé

Current self-supervised approaches for skeleton action representation learning often focus on constrained scenarios, where videos and skeleton data are recorded in laboratory settings. When dealing with estimated skeleton data in realworld videos, such methods perform poorly due to the large variations across subjects and camera viewpoints. To address this issue, we introduce ViA, a novel View-Invariant Autoencoder for self-supervised skeleton action representation learning 1. ViA leverages motion retargeting between different human performers as a pretext task, in order to disentangle the latent action-specific 'Motion' features on top of the visual representation of a 2D or 3D skeleton sequence. Such 'Motion' features are invariant to skeleton geometry and camera view and allow ViA to facilitate both, cross-subject and cross-view action classification tasks. We conduct a study focusing on transfer-learning for skeleton-based action recognition with self-supervised pre-training on real-world data (e.g., Posetics). Our results showcase that skeleton representations learned from ViA are generic enough to improve upon stateof-the-art action classification accuracy, not only on 3D laboratory datasets such as NTU-RGB+D 60 and NTU-RGB+D 120, but also on real-world datasets where only 2D data are accurately estimated, e.g., Toyota Smarthome, UAV-Human and Penn Action.
Fichier principal
Vignette du fichier
ViA__View_invariant_Skeleton_Action_Representation_Learning_via_Motion_Retargeting.pdf (11.13 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03906649 , version 1 (19-12-2022)

Identifiants

Citer

Di Yang, Yaohui Wang, Antitza Dantcheva, Lorenzo Garattoni, Gianpiero Francesca, et al.. View-invariant Skeleton Action Representation Learning via Motion Retargeting. International Journal of Computer Vision, 2024, ⟨10.1007/s11263-023-01967-8⟩. ⟨hal-03906649⟩
91 Consultations
12 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More