Logarithmic regret in communicating MDPs: Leveraging known dynamics with bandits - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Logarithmic regret in communicating MDPs: Leveraging known dynamics with bandits

Résumé

We study regret minimization in an average-reward and communicating Markov Decision Process (MDP) with known dynamics, but unknown reward function. Although learning in such MDPs is a priori easier than in fully unknown ones, they are still largely challenging as they include as special cases large classes of problems such as combinatorial semi-bandits. Leveraging the knowledge on transition function in regret minimization, in a statistically efficient way, appears largely unexplored. As it is conjectured that achieving exact optimality in generic MDPs is NP-hard, even with known transitions, we focus on a computationally efficient relaxation, at the cost of achieving order-optimal logarithmic regret instead of exact optimality. We contribute to filling this gap by introducing a novel algorithm based on the popular Indexed Minimum Empirical Divergence strategy for bandits. A key component of the proposed algorithm is a carefully designed stopping criterion leveraging the recurrent classes induced by stationary policies. We derive a nonasymptotic, problem-dependent, and logarithmic regret bound for this algorithm, which relies on a novel regret decomposition leveraging the structure. We further provide an efficient implementation and experiments illustrating its promising empirical performance.
Fichier principal
Vignette du fichier
acml_2023_fabien.pdf (1.11 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04241513 , version 1 (13-10-2023)

Identifiants

  • HAL Id : hal-04241513 , version 1

Citer

Hassan Saber, Fabien Pesquerel, Odalric-Ambrym Maillard, Mohammad Sadegh Talebi. Logarithmic regret in communicating MDPs: Leveraging known dynamics with bandits. Asian Conference on Machine Learning, Nov 2023, Istanbul, Turkey. ⟨hal-04241513⟩
39 Consultations
118 Téléchargements

Partager

Gmail Facebook X LinkedIn More