RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank

Résumé

Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations and few principled guidelines that would help practitioners to successfully deploy those methods. The main reason for that pitfall actually comes from JE-SSL's core principle of not employing any input reconstruction. Without any visual clue, it becomes extremely cryptic to judge the quality of a learned representation without having access to a labelled dataset. We hope to correct those limitations by providing a single --theoretically motivated-- criterion that reflects the quality of learned JE-SSL representations: their effective rank. Albeit simple and computationally friendly, this method --coined RankMe--- allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels, training or parameters to tune. Through thorough empirical experiments involving hundreds of repeated training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no loss in final performance compared to the current selection method that involve dataset labels. We hope that RankMe will facilitate the use of JE-SSL in domains with little or no labeled data.
Fichier principal
Vignette du fichier
main.pdf (792.04 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03793283 , version 1 (30-09-2022)
hal-03793283 , version 2 (13-02-2023)
hal-03793283 , version 3 (19-06-2023)

Identifiants

Citer

Quentin Garrido, Randall Balestriero, Laurent Najman, Yann Lecun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. 2022. ⟨hal-03793283v1⟩
95 Consultations
99 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More