Pré-Publication, Document De Travail Année : 2020

Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey

Benyamin Ghojogh
  • Fonction : Auteur
  • PersonId : 1398371
Ali Ghodsi
  • Fonction : Auteur
  • PersonId : 1398372

Résumé

This is a tutorial and survey paper on the attention mechanism, transformers, BERT, and GPT. We first explain attention mechanism, sequence-to-sequence model without and with attention, self-attention, and attention in different areas such as natural language processing and computer vision. Then, we explain transformers which do not use any recurrence. We explain all the parts of encoder and decoder in the transformer, including positional encoding, multihead self-attention and cross-attention, and masked multihead attention. Thereafter, we introduce the Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) as the stacks of encoders and decoders of transformer, respectively. We explain their characteristics and how they work.

Fichier principal
Vignette du fichier
Transformer_tutorial.pdf (1.71 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04637647 , version 1 (07-07-2024)

Licence

Identifiants

  • HAL Id : hal-04637647 , version 1

Citer

Benyamin Ghojogh, Ali Ghodsi. Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey. 2020. ⟨hal-04637647⟩
2338 Consultations
6034 Téléchargements

Partager

  • More