Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning

Reda Ouhamma
  • Fonction : Auteur
  • PersonId : 1112066
Debabrota Basu
Odalric-Ambrym Maillard

Résumé

We study the problem of episodic reinforcement learning in continuous stateaction spaces with unknown rewards and transitions. Specifically, we consider the setting where the rewards and transitions are modeled using parametric bilinear exponential families. We propose an algorithm, BEF-RLSVI, that a) uses penalized maximum likelihood estimators to learn the unknown parameters, b) injects a calibrated Gaussian noise in the parameter of rewards to ensure exploration, and c) leverages linearity of the exponential family with respect to an underlying RKHS to perform tractable planning. We further provide a frequentist regret analysis of BEF-RLSVI that yields an upper bound of Õ( (d^3 H^3 K)^{1/2} ), where d is the dimension of the parameters, H is the episode length, and K is the number of episodes. Our analysis improves the existing bounds for the bilinear exponential family of MDPs by √H and removes the handcrafted clipping deployed in existing RLSVI-type algorithms. Our regret bound is order-optimal with respect to H and K.
Fichier principal
Vignette du fichier
neurips_2022.pdf (493.93 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03790997 , version 1 (30-09-2022)
hal-03790997 , version 2 (26-10-2023)

Licence

Paternité - Pas d'utilisation commerciale

Identifiants

Citer

Reda Ouhamma, Debabrota Basu, Odalric-Ambrym Maillard. Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning. EWRL 2022 – European Workshop on Reinforcement Learning, Sep 2022, Milan, Italy. ⟨hal-03790997v1⟩
62 Consultations
31 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More