Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models

Résumé

This paper presents Words2Contact, a language-guided multi-contact placement pipeline leveraging large language models and vision language models. Our method is a key component for language-assisted teleoperation and human-robot cooperation, where human operators can instruct the robots where to place their support contacts before whole-body reaching or manipulation using natural language. Words2Contact transforms the verbal instructions of a human operator into contact placement predictions; it also deals with iterative corrections, until the human is satisfied with the contact location identified in the robot’s field of view. We benchmark state-of-the-art LLMs and VLMs for size and performance in contact prediction. We demonstrate the effectiveness of the iterative correction process, showing that users, even naive, quickly learn how to instruct the system to obtain accurate locations. Finally, we validate Words2Contact in real-world experiments with the Talos humanoid robot, instructed by human operators to place support contacts on different locations and surfaces to avoid falling when reaching for distant objects.
Fichier principal
Vignette du fichier
main.pdf (18.22 Mo) Télécharger le fichier
words2contact_video.mp4 (144.5 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04827434 , version 1 (09-12-2024)

Licence

Identifiants

Citer

Dionis Totsila, Quentin Rouxel, Jean-Baptiste Mouret, Serena Ivaldi. Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models. 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), Nov 2024, Nancy, France. pp.9-16, ⟨10.1109/Humanoids58906.2024.10769902⟩. ⟨hal-04827434⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More