A Deep Learning Framework for Recognizing both Static and Dynamic Gestures - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Sensors Année : 2021

A Deep Learning Framework for Recognizing both Static and Dynamic Gestures

Osama Mazhar
Sofiane Ramdani
Andrea Cherubini

Résumé

Intuitive user interfaces are indispensable to interact with the human centric smart environments. In this paper, we propose a unified framework that recognizes both static and dynamic gestures, using simple RGB vision (without depth sensing). This feature makes it suitable for inexpensive human-robot interaction in social or industrial settings. We employ a pose-driven spatial attention strategy, which guides our proposed Static and Dynamic gestures Network – StaDNet. From the image of the human upper body, we estimate his/her depth, along with the region-of-interest around his/her hands. The Convolutional Neural Network in StaDNet is fine-tuned on a background-substituted hand gestures dataset. It is utilized to detect 10 static gestures for each hand as well as to obtain the hand image-embeddings. These are subsequently fused with the augmented pose vector and then passed to the stacked Long Short-Term Memory blocks. Thus, human-centred frame-wise information from the augmented pose vector and from the left/right hands image-embeddings are aggregated in time to predict the dynamic gestures of the performing person. In a number of experiments, we show that the proposed approach surpasses the state-of-the-art results on the large-scale Chalearn 2016 dataset. Moreover, we transfer the knowledge learned through the proposed methodology to the Praxis gestures dataset, and the obtained results also outscore the state-of-the-art on this dataset.
Fichier principal
Vignette du fichier
J30 - mazhar -sensors 2021.pdf (2.78 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03173140 , version 1 (18-03-2021)

Licence

Paternité

Identifiants

Citer

Osama Mazhar, Sofiane Ramdani, Andrea Cherubini. A Deep Learning Framework for Recognizing both Static and Dynamic Gestures. Sensors, 2021, 21 (6), pp.#2227. ⟨10.3390/s21062227⟩. ⟨hal-03173140⟩
55 Consultations
102 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More