Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy - Archive ouverte HAL
Pré-Publication, Document De Travail Année : 2024

Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy

Résumé

Generalizing language-conditioned robotic policies to new tasks remains a significant challenge, hampered by the lack of suitable simulation benchmarks. In this paper, we address this gap by introducing GemBench, a novel benchmark to assess generalization capabilities of vision-language robotic manipulation policies. GemBench incorporates seven general action primitives and four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We evaluate state-of-the-art approaches on GemBench and also introduce a new method. Our approach 3D-LOTUS leverages rich 3D information for action prediction conditioned on language. While 3D-LOTUS excels in both efficiency and performance on seen tasks, it struggles with novel tasks. To address this, we present 3D-LOTUS++, a framework that integrates 3D-LOTUS's motion planning capabilities with the task planning capabilities of LLMs and the object grounding accuracy of VLMs. 3D-LOTUS++ achieves state-ofthe-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation. The benchmark, codes and trained models are available at https://www.di.ens.fr/willow/research/gembench/.
Fichier principal
Vignette du fichier
ICRA25_Ricardo_GemBench_arxiv.pdf (3.55 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04721101 , version 1 (04-10-2024)

Identifiants

Citer

Ricardo Garcia, Shizhe Chen, Cordelia Schmid. Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy. 2024. ⟨hal-04721101⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More