Optimistic PAC Reinforcement Learning: the Instance-Dependent View - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Optimistic PAC Reinforcement Learning: the Instance-Dependent View

Andrea Tirinzoni
  • Fonction : Auteur
  • PersonId : 1160571

Résumé

Optimistic algorithms have been extensively studied for regret minimization in episodic tabular MDPs, both from a minimax and an instance-dependent view. However, for the PAC RL problem, where the goal is to identify a near-optimal policy with high probability, little is known about their instance-dependent sample complexity. A negative result of Wagenmaker et al. (2022) suggests that optimistic sampling rules cannot be used to attain the (still elusive) optimal instance-dependent sample complexity. On the positive side, we provide the first instance-dependent bound for an optimistic algorithm for PAC RL, BPI-UCRL, for which only minimax guarantees were available (Kaufmann et al., 2021). While our bound features some minimal visitation probabilities, it also features a refined notion of sub-optimality gap compared to the value gaps that appear in prior work. Moreover, in MDPs with deterministic transitions, we show that BPI-UCRL is actually near-optimal. On the technical side, our analysis is very simple thanks to a new "target trick" of independent interest. We complement these findings with a novel hardness result explaining why the instance-dependent complexity of PAC RL cannot be easily related to that of regret minimization, unlike in the minimax regime.
Fichier principal
Vignette du fichier
bpi_ucrl.pdf (357.08 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03767409 , version 1 (01-09-2022)

Identifiants

Citer

Andrea Tirinzoni, Aymen Al-Marjani, Emilie Kaufmann. Optimistic PAC Reinforcement Learning: the Instance-Dependent View. EWRL 2022 - European Workshop on Reinforcement Learning, Sep 2022, Milan, Italy. ⟨hal-03767409⟩
46 Consultations
37 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More