Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties - Archive ouverte HAL
Conference Papers Year : 2022

Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties

Abstract

The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator. In particular, current progress in multi body dynamic simulators are underpinning the implementation of reinforcement learning for endto-end control of robotic systems. Yet simulators are mostly considered as black boxes while we have the knowledge to make them produce a richer information. In this paper, we are proposing to use the derivatives of the simulator to help with the convergence of the learning. For that, we combine model-based trajectory optimization to produce informative trials using 1st-and 2nd-order simulation derivatives. These locally-optimal runs give fair estimates of the value function and its derivatives, that we use to accelerate the convergence of the critics using Sobolev learning. We empirically demonstrate that the algorithm leads to a faster and more accurate estimation of the value function. The resulting value estimate is used in model-predictive controller as a proxy for shortening the preview horizon. We believe that it is also a first step toward superlinear reinforcement learning algorithm using simulation derivatives, that we need for end-to-end legged locomotion.
Fichier principal
Vignette du fichier
ICRA22_1504_FI (1).pdf (2.46 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03356261 , version 1 (27-09-2021)
hal-03356261 , version 2 (03-03-2022)

Identifiers

  • HAL Id : hal-03356261 , version 2

Cite

Amit Parag, Sébastien Kleff, Léo Saci, Nicolas Mansard, Olivier Stasse. Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties. International Conference on Robotics and Automation (ICRA 2022), May 2022, Philadelphia, United States. ⟨hal-03356261v2⟩
401 View
537 Download

Share

More