IMED-RL: Regret optimal learning of ergodic Markov decision processes - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

IMED-RL: Regret optimal learning of ergodic Markov decision processes

Fabien Pesquerel
  • Fonction : Auteur
  • PersonId : 1116626
Odalric-Ambrym Maillard

Résumé

We consider reinforcement learning in a discrete, undiscounted, infinite-horizon Markov Decision Problem (MDP) under the average reward criterion, and focus on the minimization of the regret with respect to an optimal policy, when the learner does not know the rewards nor the transitions of the MDP. In light of their success at regret minimization in multi-armed bandits, popular bandit strategies, such as the optimistic UCB, KL-UCB or the Bayesian Thompson sampling strategy, have been extended to the MDP setup. Despite some key successes, existing strategies for solving this problem either fail to be provably asymptotically optimal, or suffer from prohibitive burn-in phase and computational complexity when implemented in practice. In this work, we shed a novel light on regret minimization strategies, by extending to reinforcement learning the computationally appealing Indexed Minimum Empirical Divergence (IMED) bandit algorithm. Traditional asymptotic problem-dependent lower bounds on the regret are known under the assumption that the MDP is ergodic. Under this assumption, we introduce IMED-RL and prove that its regret upper bound asymptotically matches the regret lower bound. We discuss both the case when the supports of transitions are unknown, and the more informative but a priori harder-to-exploit-optimally case when they are known. Rewards are assumed light-tailed, semi-bounded from above. Last, we provide numerical illustrations on classical tabular MDPs, ergodic and communicating only, showing the competitiveness of IMED-RL in finite-time against state-of-the-art algorithms. IMED-RL also benefits from a light complexity.
Fichier principal
Vignette du fichier
IMED_RL_pesquerel_neurips.pdf (1.42 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03825423 , version 1 (22-10-2022)

Identifiants

  • HAL Id : hal-03825423 , version 1

Citer

Fabien Pesquerel, Odalric-Ambrym Maillard. IMED-RL: Regret optimal learning of ergodic Markov decision processes. NeurIPS 2022 - Thirty-sixth Conference on Neural Information Processing Systems, Nov 2022, New-Orleans, United States. ⟨hal-03825423⟩
138 Consultations
315 Téléchargements

Partager

Gmail Facebook X LinkedIn More