Learning Supplementary NLP Features for CTR Prediction in Sponsored Search - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Learning Supplementary NLP Features for CTR Prediction in Sponsored Search

Yunqing Xia
  • Fonction : Auteur
  • PersonId : 1287788
Weiwei Deng
  • Fonction : Auteur
  • PersonId : 1287789
Qi Zhang
  • Fonction : Auteur
  • PersonId : 1287790

Résumé

In sponsored search engines, pre-trained language models have shown promising performance improvements on Click-Through-Rate (CTR) prediction. A widely used approach for utilizing pretrained language models in CTR prediction consists of fine-tuning the language models with click labels and early stopping on peak value of the obtained Area Under the ROC Curve (AUC). Thereafter the output of these fine-tuned models, i.e., the final score or intermediate embedding generated by language model, is used as a new Natural Language Processing (NLP) feature into CTR prediction baseline. This cascade approach avoids complicating the CTR prediction baseline, while keeping flexibility and agility. However, we show in this work that calibrating separately the language model based on the peak single model AUC does not always yield NLP features that give the best performance in CTR prediction model ultimately. Our analysis reveals that the misalignment is due to overlap and redundancy between the new NLP features and the existing features in CTR prediction baseline. In other words, the NLP features can improve CTR prediction better if such overlap can be reduced. For this purpose, we introduce a simple and general joint-training framework for fine-tuning of language models, combined with the already existing features in CTR prediction baseline, to extract supplementary knowledge for NLP feature. Moreover, we develop an efficient Supplementary Knowledge Distillation (SuKD) that transfers the supplementary knowledge learned by a heavy language model to a light and serviceable model. Comprehensive experiments on both public data and commercial data presented in this work demonstrate that the new NLP features resulting from the jointtraining framework can outperform significantly the ones from the independent fine-tuning based on click labels. we also show that the light model distilled with SuKD can provide obvious AUC
Fichier principal
Vignette du fichier
3534678.3539064-3.pdf (1.23 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04219734 , version 1 (27-09-2023)

Identifiants

Citer

Dong Wang, Shaoguang Yan, Yunqing Xia, Kavé Salamatian, Weiwei Deng, et al.. Learning Supplementary NLP Features for CTR Prediction in Sponsored Search. KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Aug 2022, Washington DC USA, United States. pp.4010-4020, ⟨10.1145/3534678.3539064⟩. ⟨hal-04219734⟩

Collections

UNIV-SAVOIE LISTIC
14 Consultations
35 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More