Offline Reinforcement Learning as Anti-Exploration - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Offline Reinforcement Learning as Anti-Exploration

Résumé

Offline Reinforcement Learning (RL) aims at learning an optimal control from a fixed dataset, without interactions with the system. An agent in this setting should avoid selecting actions whose consequences cannot be predicted from the data. This is the converse of exploration in RL, which favors such actions. We thus take inspiration from the literature on bonus-based exploration to design a new offline RL agent. The core idea is to subtract a prediction-based exploration bonus from the reward, instead of adding it for exploration. This allows the policy to stay close to the support of the dataset. We connect this approach to a more common regularization of the learned policy towards the data. Instantiated with a bonus based on the prediction error of a variational autoencoder, we show that our agent is competitive with the state of the art on a set of continuous control locomotion and manipulation tasks.

Dates et versions

hal-03468875 , version 1 (07-12-2021)

Identifiants

Citer

Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, et al.. Offline Reinforcement Learning as Anti-Exploration. AAAI 2022 - 36th AAAI Conference on Artificial Intelligence, Association for the Advancement of Artificial Intelligence (AAAI), Feb 2022, Vancouver, Canada. ⟨hal-03468875⟩
108 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More