LightGBM: A Highly Efficient Gradient Boosting Decision Tree - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2017

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Guolin Ke
  • Fonction : Auteur
  • PersonId : 1218804
Thomas Finley
  • Fonction : Auteur
Taifeng Wang
  • Fonction : Auteur
  • PersonId : 1218805
Qiwei Ye
  • Fonction : Auteur
Tie-Yan Liu
  • Fonction : Auteur
  • PersonId : 1218806

Résumé

Gradient Boosting Decision Tree (GBDT) is a popular machine learning algorithm, and has quite a few effective implementations such as XGBoost and pGBRT. Although many engineering optimizations have been adopted in these implementations, the efficiency and scalability are still unsatisfactory when the feature dimension is high and data size is large. A major reason is that for each feature, they need to scan all the data instances to estimate the information gain of all possible split points, which is very time consuming. To tackle this problem, we propose two novel techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). With GOSS, we exclude a significant proportion of data instances with small gradients, and only use the rest to estimate the information gain. We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size. With EFB, we bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously), to reduce the number of features. We prove that finding the optimal bundling of exclusive features is NP-hard, but a greedy algorithm can achieve quite good approximation ratio (and thus can effectively reduce the number of features without hurting the accuracy of split point determination by much). We call our new GBDT implementation with GOSS and EFB LightGBM. Our experiments on multiple public datasets show that, LightGBM speeds up the training process of conventional GBDT by up to over 20 times while achieving almost the same accuracy.
Fichier principal
Vignette du fichier
Ke2017.pdf (209.86 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03953007 , version 1 (23-01-2023)

Licence

Identifiants

  • HAL Id : hal-03953007 , version 1

Citer

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, et al.. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. 31st International Conference on Neural Information Processing Systems, Dec 2017, Long Beach, United States. ⟨hal-03953007⟩
96 Consultations
400 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More