Revisiting Peng's Q(λ) for for modern reinforcement learning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Revisiting Peng's Q(λ) for for modern reinforcement learning

Tadashi Kozuno
  • Fonction : Auteur
  • PersonId : 1105627
Mark Rowland
  • Fonction : Auteur
Rémi Munos
  • Fonction : Auteur
Will Dabney
  • Fonction : Auteur
Michal Valko
David Abel
  • Fonction : Auteur

Résumé

Off-policy multi-step reinforcement learning algorithms consist of conservative and nonconservative algorithms: the former actively cut traces, whereas the latter do not. Recently, Munos et al. (2016) proved the convergence of conservative algorithms to an optimal Q-function. In contrast, non-conservative algorithms are thought to be unsafe and have a limited or no theoretical guarantee. Nonetheless, recent studies have shown that non-conservative algorithms empirically outperform conservative ones. Motivated by the empirical results and the lack of theory, we carry out theoretical analyses of Peng's Q(λ), a representative example of non-conservative algorithms. We prove that it also converges to an optimal policy provided that the behavior policy slowly tracks a greedy policy in a way similar to conservative policy iteration. Such a result has been conjectured to be true but has not been proven. We also experiment with Peng's Q(λ) in complex continuous control tasks, confirming that Peng's Q(λ) often outperforms conservative algorithms despite its simplicity. These results indicate that Peng's Q(λ), which was thought to be unsafe, is a theoretically-sound and practically effective algorithm.
Fichier principal
Vignette du fichier
kozuno2021revisiting.pdf (3.28 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03289292 , version 1 (16-07-2021)

Identifiants

  • HAL Id : hal-03289292 , version 1

Citer

Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, et al.. Revisiting Peng's Q(λ) for for modern reinforcement learning. International Conference on Machine Learning, Jul 2021, Vienna / Virtual, Austria. ⟨hal-03289292⟩
27 Consultations
52 Téléchargements

Partager

Gmail Facebook X LinkedIn More