TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Pattern Analysis and Machine Intelligence Année : 2023

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut

Résumé

In this paper, we describe a graph-based algorithm that uses the features obtained by a self-supervised transformer to detect and segment salient objects in images and videos. With this approach, the image patches that compose an image or video are organised into a fully connected graph, where the edge between each pair of patches is labeled with a similarity score between patches using features learned by the transformer. Detection and segmentation of salient objects is then formulated as a graph-cut problem and solved using the classical Normalized Cut algorithm. Despite the simplicity of this approach, it achieves state-of-the-art results on several common image and video detection and segmentation tasks. For unsupervised object discovery, this approach outperforms the competing approaches by a margin of 6.1%, 5.7%, and 2.6%, respectively, when tested with the VOC07, VOC12, and COCO20K datasets. For the unsupervised saliency detection task in images, this method improves the score for Intersection over Union (IoU) by 4.4%, 5.6% and 5.2%. When tested with the ECSSD, DUTS, and DUT-OMRON datasets, respectively, compared to current state-of-the-art techniques. This method also achieves competitive results for unsupervised video object segmentation tasks with the DAVIS, SegTV2, and FBMS datasets.
Fichier principal
Vignette du fichier
TokenCutVideo.pdf (8.99 Mo) Télécharger le fichier
TokenCutVideo (16.17 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03765422 , version 1 (31-08-2022)
hal-03765422 , version 2 (09-09-2022)
hal-03765422 , version 3 (30-11-2023)

Identifiants

Citer

Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, et al.. TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (12), pp.15790 - 15801. ⟨10.1109/TPAMI.2023.3305122⟩. ⟨hal-03765422v3⟩
145 Consultations
63 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More