Asymmetric multi-task learning for interpretable gaze-driven grasping action forecasting - Archive ouverte HAL
Article Dans Une Revue IEEE Journal of Biomedical and Health Informatics Année : 2024

Asymmetric multi-task learning for interpretable gaze-driven grasping action forecasting

Résumé

This work tackles the problem of automatically predicting the grasping intention of humans observing their environment, with eye-tracker glasses and video cameras recording the scene view. Our target application is the assistance to people with motor disabilities and potential cognitive impairments, using assistive robotics. Our proposal leverages the analysis of human attention captured in the form of gaze fixations recorded by an eye-tracker on the first person video, as the anticipation of prehension actions is a well studied and well known phenomenon. We propose a multi-task system that simultaneously addresses the prediction of human attention in the near future, and the anticipation of grasping actions. In our model, visual attention is modeled as a competitive process between a discrete set of states, each one associated to a well-known gaze movement pattern from visual psychology. We additionally consider an asymmetric multitask problem, where attention modeling is an auxiliary task that helps to regularize the learning process of the main action prediction task, and propose a constrained multi-task loss that naturally deals with this asymmetry. Our model shows superior performance than other losses for dynamic multi-task learning, current dominant deep architectures for general action forecasting and particularly-tailored models for predicting grasping intention. In particular, it provides state-of-the-art performance in three datasets for egocentric action anticipation, with an average precision of 0.569 and 0.524 in GITW and Sharon datasets, respectively, and an accuracy of 89.2% and a success rate of 51.7% in Invisible dataset.

Fichier principal
Vignette du fichier
Gonzalez-Dıaz_etal_IEEE_JBHI_2024.pdf (5.26 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04727461 , version 1 (09-10-2024)

Identifiants

Citer

Iván González-Diaz, Miguel Molina-Moreno, Jenny Benois-Pineau, Aymar de Rugy. Asymmetric multi-task learning for interpretable gaze-driven grasping action forecasting. IEEE Journal of Biomedical and Health Informatics, 2024, pp.1 - 17. ⟨10.1109/jbhi.2024.3430810⟩. ⟨hal-04727461⟩

Collections

CNRS
0 Consultations
0 Téléchargements

Altmetric

Partager

More