Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation

Imad Aouali
  • Fonction : Auteur
Amine Benhalloum
  • Fonction : Auteur
Martin Bompaire
  • Fonction : Auteur
Benjamin Heymann
  • Fonction : Auteur
David Rohde
  • Fonction : Auteur
Otmane Sakhi
  • Fonction : Auteur
Flavian Vasile
  • Fonction : Auteur

Résumé

Both in academic and industry-based research, online evaluation methods are seen as the golden standard for interactive applications like recommendation systems. Naturally, the reason for this is that we can directly measure utility metrics that rely on interventions, being the recommendations that are being shown to users. Nevertheless, online evaluation methods are costly for a number of reasons, and a clear need remains for reliable offline evaluation procedures. In industry, offline metrics are often used as a first-line evaluation to generate promising candidate models to evaluate online. In academic work, limited access to online systems makes offline metrics the de facto approach to validating novel methods. Two classes of offline metrics exist: proxy-based methods, and counterfactual methods. The first class is often poorly correlated with the online metrics we care about, and the latter class only provides theoretical guarantees under assumptions that cannot be fulfilled in real-world environments. Here, we make the case that simulation-based comparisons provide ways forward beyond offline metrics, and argue that they are a preferable means of evaluation.
Fichier principal
Vignette du fichier
position_paper_simu.pdf (124.64 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03959853 , version 1 (27-01-2023)

Identifiants

Citer

Imad Aouali, Amine Benhalloum, Martin Bompaire, Benjamin Heymann, Olivier Jeunen, et al.. Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation. ACM RecSys 2021 Workshop on Simulation Methods for Recommender Systems, Sep 2021, Amsterdam, Netherlands. ⟨hal-03959853⟩
5 Consultations
18 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More