What is the Role of Small Models in the LLM Era: A Survey - Archive ouverte HAL
Pré-Publication, Document De Travail Année : 2024

What is the Role of Small Models in the LLM Era: A Survey

Résumé

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models
Fichier principal
Vignette du fichier
a_survey_of_the_role_of_small_models.pdf (319.76 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04712720 , version 1 (27-09-2024)

Licence

Identifiants

Citer

Lihu Chen, Gaël Varoquaux. What is the Role of Small Models in the LLM Era: A Survey. 2024. ⟨hal-04712720⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More