Training data-efficient image transformers & distillation through attention - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Training data-efficient image transformers & distillation through attention

Touvron Hugo
  • Fonction : Auteur
Matthieu Cord
Douze Matthijs
  • Fonction : Auteur
Massa Francisco
  • Fonction : Auteur
Sablayrolles Alexandre
  • Fonction : Auteur
Jegou Herve
  • Fonction : Auteur

Résumé

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. These highperforming vision transformers are pre-trained with hundreds of millions of images using a large infrastructure, thereby limiting their adoption. In this work, we produce competitive convolutionfree transformers trained on ImageNet only using a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop) on ImageNet with no external data. We also introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention, typically from a convnet teacher. The learned transformers are competitive (85.2% top-1 acc.) with the state of the art on ImageNet, and similarly when transferred to other tasks. We will share our code and models.
Fichier non déposé

Dates et versions

hal-03997937 , version 1 (20-02-2023)

Identifiants

  • HAL Id : hal-03997937 , version 1

Citer

Touvron Hugo, Matthieu Cord, Douze Matthijs, Massa Francisco, Sablayrolles Alexandre, et al.. Training data-efficient image transformers & distillation through attention. ICML, Jul 2021, visio, France. ⟨hal-03997937⟩
719 Consultations
0 Téléchargements

Partager

Gmail Facebook X LinkedIn More