RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving

Angelika Ando
  • Fonction : Auteur
Spyros Gidaris
  • Fonction : Auteur
Andrei Bursuc
  • Fonction : Auteur
Gilles Puy
  • Fonction : Auteur
Alexandre Boulch
  • Fonction : Auteur

Résumé

Casting semantic segmentation of outdoor LiDAR point clouds as a 2D problem, e.g., via range projection, is an effective and popular approach. These projection-based methods usually benefit from fast computations and, when combined with techniques which use other point cloud representations, achieve state-of-the-art results. Today, projection-based methods leverage 2D CNNs but recent advances in computer vision show that vision transformers (ViTs) have achieved state-of-the-art results in many image- based benchmarks. In this work, we question if projection- based methods for 3D semantic segmentation can benefit from these latest improvements on ViTs. We answer positively but only after combining them with three key ingredients: (a) ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. By preserving the same backbone architecture as for RGB images, we can exploit the knowledge from long training on large image collections that are much cheaper to acquire and annotate than point clouds. We reach our best results with pre-trained ViTs on large image datasets. (b) We compensate ViTs' lack of inductive bias by substituting a tailored convolutional stem for the classical linear embedding layer. (c) We refine pixel-wise predictions with a convolutional decoder and a skip connection from the convolutional stem to combine low-level but fine-grained features of the the convolutional stem with the high-level but coarse predictions of the ViT encoder. With these ingredients, we show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and SemanticKITTI. The code is available at https://github.com/valeoai/rangevit.
Fichier principal
Vignette du fichier
Ando_RangeViT_Towards_Vision_Transformers_for_3D_Semantic_Segmentation_in_Autonomous_CVPR_2023_paper.pdf (1.24 Mo) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04496524 , version 1 (08-03-2024)

Identifiants

Citer

Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, et al.. RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), Jun 2023, Vancouver, Canada. pp.5240-5250, ⟨10.1109/CVPR52729.2023.00507⟩. ⟨hal-04496524⟩
19 Consultations
1 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More