A Performance Study of LLM-Generated Code on Leetcode - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

A Performance Study of LLM-Generated Code on Leetcode

Résumé

This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. This research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, irrespective of the adopted LLM. We also find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans. The paper further discusses the use of Leetcode as a benchmarking dataset, the limita- tions imposed by potential data contamination, and the platform’s measurement reliability. We believe that our findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field.
Fichier principal
Vignette du fichier
Tristan___LLM_Code_Performance_Study.pdf (1.14 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04525620 , version 1 (28-03-2024)

Licence

Identifiants

Citer

Tristan Coignion, Clément Quinton, Romain Rouvoy. A Performance Study of LLM-Generated Code on Leetcode. EASE'24 - 28th International Conference on Evaluation and Assessment in Software Engineering, Jun 2024, Salerno, Italy. ⟨10.1145/3661167.3661221⟩. ⟨hal-04525620⟩
748 Consultations
1034 Téléchargements

Altmetric

Partager

More