Dynamic allocation optimization in A/B tests using classification-based preprocessing
Résumé
An A/B test evaluates the impact of a new technology by running it in a real production environment and testing its performance on a set of items. Recently, promising new methods are optimizing A/B tests with dynamic allocation. They allow for a quicker result regarding which variation (A or B) is the best, saving money for the user. However, dynamic allocation by traditional methods requires certain assumptions, which are not always verified in reality. This is mainly due to the fact that the populations tested are not homogeneous. This document reports on the new reinforcement learning methodology which has been deployed by the commercial A/B testing platform AB Tasty. We provide a new method that not only builds homogeneous groups for a user, but also allows to find the best variation for these groups in a short period of time. This paper provides numerical results on AB Tasty data, but also on public data sets, to demonstrate an improvement in A/B testing over traditional methods.
Origine : Fichiers produits par l'(les) auteur(s)