Early gesture detection in untrimmed streams: A controlled CTC approach for reliable decision-making - Archive ouverte HAL
Article Dans Une Revue Pattern Recognition Année : 2024

Early gesture detection in untrimmed streams: A controlled CTC approach for reliable decision-making

Résumé

This paper focuses on the problem of online action detection for interactive systems, with a special emphasis on earliness. Online Action Detection (OAD) refers to the challenging task of recognizing gestures in untrimmed, streaming videos where the actions occur in unpredictable orders and durations. To address these challenges, we present a skeleton-based system for OAD incorporating a decision mechanism to accurately detect ongoing gestures. This allows us to provide instance-level output, achieving a high level of stream understanding. This mechanism relies on a novel Connectionist Temporal Classification (CTC) loss design that restricts the path possibilities according to the action boundaries. We also present a mechanism to tune the trade-off between accuracy and earliness according to the needs of the interactive system using a weighted label prior. This system includes a 3D CNN network, referred to as DOLT-C3D, exploiting the spatial-temporal information provided by the euclidean skeleton representation. We extensively evaluate our approach on eight publicly available datasets, demonstrating its superior performance compared to state-of-the-art methods in terms of both accuracy and earliness. We also successfully applied our approach to early 2D gestures detection. Furthermore, our system shows real-time performance, making it a suitable choice for interactive systems.
Fichier principal
Vignette du fichier
Article_PR_preprintfinal.pdf (3.67 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04634678 , version 1 (04-07-2024)

Licence

Identifiants

Citer

William Mocaër, Eric Anquetil, Richard Kulpa. Early gesture detection in untrimmed streams: A controlled CTC approach for reliable decision-making. Pattern Recognition, 2024, pp.110733. ⟨10.1016/j.patcog.2024.110733⟩. ⟨hal-04634678⟩
43 Consultations
37 Téléchargements

Altmetric

Partager

More