Communication Dans Un Congrès Année : 2025

BMPipe: Bubble-Memory Co-Optimization Strategy Planner for Very-Large DNN Training

Thibaut Tachon
  • Fonction : Auteur
  • PersonId : 1661263
Raja Appuswamy
Teng Su
  • Fonction : Auteur
  • PersonId : 1661264

Résumé

Pipeline parallelism and activation recomputation are widely adopted optimization techniques, among others, to scale DNN training on large accelerator clusters. However, as DNNs grow in complexity and heterogeneity, it becomes increasingly difficult to determine the optimal combination of pipeline partitioning and recomputation strategies. Existing solutions either propose manual optimization approaches that do not scale or automated approaches that explore only a subset of optimization possibilities due to an explosion of search space. In this paper, we present BMPipe, a bubble-memory co-optimization planner that holistically optimizes computation imbalance, memory under utilization, redundant computation, and schedulinginduced preparation time. At its core, BMPipe uses symbolic representations that unify computation, memory, and bubbles into a single model that is solved by using an ILP-based planner. Using BMPipe, we perform a thorough experimental evaluation where we train several large, state-of-the-art DNN models on a 16K-NPU cluster. We show that BMPipe achieves up to 1.36× speedup compared to the state-of-the-art solution Megatron. Against automatic planners PipeDream, Merak and AdaPipe, it yields as 1.27× speed-up. In addition, BMPipe boosts peak device-memory utilization by 1.42× compared with Megatron.

Fichier principal
Vignette du fichier
BMPipe.pdf (1.3 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05564425 , version 1 (24-03-2026)

Licence

Identifiants

Citer

Ruiwen Wang, Chong Li, Thibaut Tachon, Raja Appuswamy, Teng Su. BMPipe: Bubble-Memory Co-Optimization Strategy Planner for Very-Large DNN Training. 2025 IEEE International Conference on Cluster Computing (CLUSTER 2025), Sep 2025, Edimbourg, United Kingdom. ⟨10.1109/CLUSTER59342.2025.11186487⟩. ⟨hal-05564425⟩
31 Consultations
42 Téléchargements

Altmetric

Partager

  • More