Self-supervised learning with rotation-invariant kernels - Archive ouverte HAL Access content directly
Preprints, Working Papers, ... Year :

Self-supervised learning with rotation-invariant kernels

Gilles Puy
Patrick Pérez
  • Function : Author


We introduce a regularization loss based on kernel mean embeddings with rotation-invariant kernels on the hypersphere (also known as dot-product kernels) for self-supervised learning of image representations. Besides being fully competitive with the state of the art, our method significantly reduces time and memory complexity for self-supervised training, making it implementable for very large embedding dimensions on existing devices and more easily adjustable than previous methods to settings with limited resources. Our work follows the major paradigm where the model learns to be invariant to some predefined image transformations (cropping, blurring, color jittering, etc.), while avoiding a degenerate solution by regularizing the embedding distribution. Our particular contribution is to propose a loss family promoting the embedding distribution to be close to the uniform distribution on the hypersphere, with respect to the maximum mean discrepancy pseudometric. We demonstrate that this family encompasses several regularizers of former methods, including uniformity-based and information-maximization methods, which are variants of our flexible regularization loss with different kernels. Beyond its practical consequences for state-of-the-art self-supervised learning with limited resources, the proposed generic regularization approach opens perspectives to leverage more widely the literature on kernel methods in order to improve self-supervised learning methods.
Fichier principal
Vignette du fichier
main.pdf (421.76 Ko) Télécharger le fichier
Vignette du fichier
mmd-vignette-bis.png (108.01 Ko) Télécharger le fichier
Vignette du fichier
mmd-vignette.png (108.01 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Format : Figure, Image
Format : Figure, Image

Dates and versions

hal-03738466 , version 1 (26-07-2022)
hal-03738466 , version 2 (03-10-2022)
hal-03738466 , version 3 (11-10-2022)
hal-03738466 , version 4 (06-03-2023)



Léon Zheng, Gilles Puy, Elisa Riccietti, Patrick Pérez, Rémi Gribonval. Self-supervised learning with rotation-invariant kernels. 2022. ⟨hal-03738466v3⟩



119 View
132 Download



Gmail Facebook Twitter LinkedIn More