Learning Compositional Neural Programs for Continuous Control - Archive ouverte HAL Access content directly
Preprints, Working Papers, ... Year :

Learning Compositional Neural Programs for Continuous Control

Thomas Pierrot
  • Function : Author
Nicolas Perrin
Alexandre Laterre
  • Function : Author
Olivier Sigaud
Karim Beguir
  • Function : Author


We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple levels of abstraction. The key insight is that the self-models enable planning by imagination, obviating the need for interaction with the world when learning higher-level compositional programs. To accomplish the third stage of learning, we extend the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural programmer-interpreters. We empirically show that AlphaNPI-X can effectively learn to tackle challenging sparse manipulation tasks, such as stacking multiple blocks, where powerful model-free baselines fail.

Dates and versions

hal-03083161 , version 1 (18-12-2020)



Thomas Pierrot, Nicolas Perrin, Feryal Behbahani, Alexandre Laterre, Olivier Sigaud, et al.. Learning Compositional Neural Programs for Continuous Control. 2020. ⟨hal-03083161⟩


29 View
0 Download



Gmail Facebook Twitter LinkedIn More