Tightening Exploration in Upper Confidence Reinforcement Learning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Tightening Exploration in Upper Confidence Reinforcement Learning

Hippolyte Bourel
  • Fonction : Auteur
  • PersonId : 1081378

Résumé

The upper confidence reinforcement learning (UCRL2) algorithm introduced in (Jaksch et al., 2010) is a popular method to perform regret minimization in unknown discrete Markov Decision Processes under the average-reward criterion. Despite its nice and generic theoretical regret guarantees , this algorithm and its variants have remained until now mostly theoretical as numerical experiments in simple environments exhibit long burn-in phases before the learning takes place. In pursuit of practical efficiency, we present UCRL3, following the lines of UCRL2, but with two key modifications: First, it uses state-of-the-art time-uniform concentration inequalities to compute confidence sets on the reward and (component-wise) transition distributions for each state-action pair. Furthermore , to tighten exploration, it uses an adap-tive computation of the support of each transition distribution, which in turn enables us to revisit the extended value iteration procedure of UCRL2 to optimize over distributions with reduced support by disregarding low probability transitions, while still ensuring near-optimism. We demonstrate , through numerical experiments in standard environments, that reducing exploration this way yields a substantial numerical improvement compared to UCRL2 and its variants. On the theoretical side, these key modifications enable us to derive a regret bound for UCRL3 improving on UCRL2, that for the first time makes appear notions of local diameter and local effective support, thanks to variance-aware concentration bounds.
Fichier principal
Vignette du fichier
ICML2020_UCRL3_FinalVersion.pdf (1.38 Mo) Télécharger le fichier
ICML2020_UCRL3_FinalVersion.synctex.gz (390.77 Ko) Télécharger le fichier
RS_ICML2020.pdf (8.28 Ko) Télécharger le fichier
RS_ICML2020.pdf_tex (6.09 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-03000664 , version 1 (12-11-2020)

Identifiants

  • HAL Id : hal-03000664 , version 1

Citer

Hippolyte Bourel, Odalric-Ambrym Maillard, Mohammad Sadegh Talebi. Tightening Exploration in Upper Confidence Reinforcement Learning. International Conference on Machine Learning, Jul 2020, Vienna, Austria. ⟨hal-03000664⟩
159 Consultations
182 Téléchargements

Partager

Gmail Facebook X LinkedIn More