Lightweight integration of 3D features to improve 2D image segmentation - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Computers and Graphics Année : 2023

Lightweight integration of 3D features to improve 2D image segmentation

Olivier Pradelle
Raphaëlle Chaine
David Wendland
  • Fonction : Auteur
Julie Digne

Résumé

Scene understanding has made tremendous progress over the past few years, as data acquisition systems are now providing an increasing amount of data of various modalities (point cloud, depth, RGB...). However, this improvement comes at a large cost on computation resources and data annotation requirements. To analyze geometric information and images jointly, many approaches rely on both a 2D loss and 3D loss, requiring not only 2D per pixel-labels but also 3D per-point labels. However, obtaining a 3D groundtruth is challenging, time-consuming and error-prone. In this paper, we show that image segmentation can benefit from 3D geometric information without requiring a 3D groundtruth, by training the geometric feature extraction and the 2D segmentation network jointly, in an end-to-end fashion, using only the 2D segmentation loss. Our method starts by extracting a map of 3D features directly from a provided point cloud by using a lightweight 3D neural network. The 3D feature map, merged with the RGB image, is then used as an input to a classical image segmentation network. Our method can be applied to many 2D segmentation networks, improving significantly their performance with only a marginal network weight increase and light input dataset requirements, since no 3D groundtruth is required.
Fichier principal
Vignette du fichier
Lightweight_integration_of_3D_features_to_improve_2D_image_segmentation.pdf (4.43 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY SA - Paternité - Partage selon les Conditions Initiales

Dates et versions

hal-04159883 , version 1 (12-07-2023)

Licence

Paternité - Partage selon les Conditions Initiales

Identifiants

Citer

Olivier Pradelle, Raphaëlle Chaine, David Wendland, Julie Digne. Lightweight integration of 3D features to improve 2D image segmentation. Computers and Graphics, 2023, 114, pp.326-336. ⟨10.1016/j.cag.2023.06.004⟩. ⟨hal-04159883⟩
32 Consultations
52 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More