Communication Dans Un Congrès Année : 2025

Online Episodic Convex Reinforcement Learning

Résumé

We study online learning in episodic finite-horizon Markov decision processes (MDPs) with convex objective functions, known as the concave utility reinforcement learning (CURL) problem. This setting generalizes RL from linear to convex losses on the state-action distribution induced by the agent’s policy. The non-linearity of CURL invalidates classical Bellman equations and requires new algorithmic approaches. We introduce the first algorithm achieving near-optimal regret bounds for online CURL without any prior knowledge on the transition function. To achieve this, we use an online mirror descent algorithm with varying constraint sets and a carefully designed exploration bonus. We then address for the first time a bandit version of CURL, where the only feedback is the value of the objective function on the state-action distribution induced by the agent's policy. We achieve a sub-linear regret bound for this more challenging problem by adapting techniques from bandit convex optimization to the MDP setting.

Fichier principal
Vignette du fichier
icml.pdf (978.61 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05059067 , version 1 (07-05-2025)

Licence

Identifiants

  • HAL Id : hal-05059067 , version 1

Citer

Bianca Marin Moreno, Khaled Eldowa, Pierre Gaillard, Margaux Brégère, Nadia Oudjane. Online Episodic Convex Reinforcement Learning. ICML 2025 - Proceedings of the 42nd International Conference on Machine Learning, Jul 2025, Vancouver, Canada. pp.44775--44824. ⟨hal-05059067⟩
399 Consultations
328 Téléchargements

Partager

  • More