Tactical Optimism and Pessimism for Deep Reinforcement Learning - Archive ouverte HAL Access content directly
Conference Papers Year : 2021

Tactical Optimism and Pessimism for Deep Reinforcement Learning

Jack Parker-Holder
  • Function : Author
  • PersonId : 1118668
Aldo Pacchiano
  • Function : Author
  • PersonId : 1118669
Michael I Jordan
  • Function : Author
  • PersonId : 1118677

Abstract

In recent years, deep off-policy actor-critic algorithms have become a dominant approach to reinforcement learning for continuous control. One of the primary drivers of this improved performance is the use of pessimistic value updates to address function approximation errors, which previously led to disappointing performance. However, a direct consequence of pessimism is reduced exploration, running counter to theoretical support for the efficacy of optimism in the face of uncertainty. So which approach is best? In this work, we show that the most effective degree of optimism can vary both across tasks and over the course of learning. Inspired by this insight, we introduce a novel deep actor-critic framework, Tactical Optimistic and Pessimistic (TOP) estimation, which switches between optimistic and pessimistic value learning online. This is achieved by formulating the selection as a multi-arm bandit problem. We show in a series of continuous control tasks that TOP outperforms existing methods which rely on a fixed degree of optimism, setting a new state of the art in challenging pixel-based environments. Since our changes are simple to implement, we believe these insights can easily be incorporated into a multitude of off-policy algorithms.
Fichier principal
Vignette du fichier
NeurIPS-2021-tactical-optimism-and-pessimism-for-deep-reinforcement-learning-Paper.pdf (1.87 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03455481 , version 1 (29-11-2021)

Identifiers

  • HAL Id : hal-03455481 , version 1

Cite

Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, Michael I Jordan. Tactical Optimism and Pessimism for Deep Reinforcement Learning. NeurIPS 2021 - Thirty-fifth Annual Conference on Neural Information Processing Systems, Dec 2021, Online, France. pp.1-15. ⟨hal-03455481⟩
81 View
79 Download

Share

Gmail Facebook X LinkedIn More