Communication Dans Un Congrès Année : 2026

Bayesian and Partition-Based Optimization for Hyperparameter Optimization of LLM Fine-Tuning

Résumé

Large Language Models (LLMs) have revolutionized natural language processing (NLP) by achieving state-of-the-art performance across diverse tasks. However, fine-tuning these models for domain-specific applications is significantly constrained by the computational costs associated with their training. In this paper, we propose two complementary approaches to address the Hyperparameter Optimization (HPO) challenge in LLM fine-tuning: Bayesian Optimization based on Gaussian Process (BO-GP) and Partition-Based Optimization (PBO). On the one hand, BO efficiently exploits historical knowledge to achieve optimal results within a limited number of evaluations, but its inherently sequential nature poses scalability challenges. On the other hand, PBO enables massive parallelization, making it more scalable but requiring significantly more evaluations to converge. To leverage their complementary strengths for optimizing expensive objective functions, we investigate these methods and propose a hybrid BO-PBO algorithm. This work represents a foundational step toward harnessing the potential of parallel Bayesian Optimization-based algorithms for solving expensive optimization problems in exascale computing environments.

Fichier non déposé

Dates et versions

hal-05464629 , version 1 (19-01-2026)

Identifiants

Citer

Nathan Davouse, El-Ghazali Talbi. Bayesian and Partition-Based Optimization for Hyperparameter Optimization of LLM Fine-Tuning. OLA2025 - International Conference in Optimization and Learning, Apr 2025, Dubai, United Arab Emirates. pp.16-33, ⟨10.1007/978-3-032-13589-6_2⟩. ⟨hal-05464629⟩
24 Consultations
0 Téléchargements

Altmetric

Partager

  • More