Reinforcement Learning with History-Dependent Dynamic Contexts - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Reinforcement Learning with History-Dependent Dynamic Contexts

Résumé

We introduce Dynamic Contextual Markov Decision Processes (DCMDPs), a novel reinforcement learning framework for history-dependent environments that generalizes the contextual MDP framework to handle non-Markov environments, where contexts change over time. We consider special cases of the model, with a focus on logistic DCMDPs, which break the exponential dependence on history length by leveraging aggregation functions to determine context transitions. This special structure allows us to derive an upper-confidence-bound style algorithm for which we establish regret bounds. Motivated by our theoretical results, we introduce a practical model-based algorithm for logistic DCMDPs that plans in a latent space and uses optimism over history-dependent features. We demonstrate the efficacy of our approach on a recommendation task (using MovieLens data) where user behavior dynamics evolve in response to recommendations.
Fichier principal
Vignette du fichier
2302.02061.pdf (1.24 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04420115 , version 1 (26-01-2024)

Identifiants

Citer

Guy Tennenholtz, Nadav Merlis, Lior Shani, Martin Mladenov, Craig Boutilier. Reinforcement Learning with History-Dependent Dynamic Contexts. ICML, 2023, Honolulu, United States. ⟨hal-04420115⟩
38 Consultations
19 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More