DIVERSITY POLICY GRADIENT FOR SAMPLE EFFI-CIENT QUALITY-DIVERSITY OPTIMIZATION - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

DIVERSITY POLICY GRADIENT FOR SAMPLE EFFI-CIENT QUALITY-DIVERSITY OPTIMIZATION

Thomas Pierrot
Valentin Macé
  • Fonction : Auteur
  • PersonId : 1156977
Felix Chalumeau
  • Fonction : Auteur
  • PersonId : 1156978
Arthur Flajolet
  • Fonction : Auteur
  • PersonId : 1156979
Geoffrey Cideron
  • Fonction : Auteur
Karim Beguir
  • Fonction : Auteur
Antoine Cully
  • Fonction : Auteur
  • PersonId : 980996
Olivier Sigaud

Résumé

A fascinating aspect of nature lies in its ability to produce a large and diverse collection of high-performing organisms in an open-ended way. By contrast, most AI algorithms seek convergence and focus on finding a single efficient solution to a given problem. Aiming for diversity through divergent search in addition to performance is a convenient way to deal with the exploration-exploitation tradeoff that plays a central role in learning. It also allows for increased robustness when the returned collection contains several working solutions to the considered problem, making it well-suited for real applications such as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed for this purpose. This paper proposes a novel algorithm, QD-PG, which combines the strength of Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performing neural policies in continuous control environments. The main contribution of this work is the introduction of a Diversity Policy Gradient (DPG) that drives policies towards more diversity in a sampleefficient and open-ended manner. Specifically, QD-PG selects neural controllers from a MAP-ELITES grid and uses two gradient-based mutation operators to improve both quality and diversity. Our results demonstrate that QD-PG is significantly more sample-efficient than its evolutionary competitors.
Fichier principal
Vignette du fichier
diversity_policy_gradient_for_.pdf (5 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03753541 , version 1 (18-08-2022)

Identifiants

  • HAL Id : hal-03753541 , version 1

Citer

Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, et al.. DIVERSITY POLICY GRADIENT FOR SAMPLE EFFI-CIENT QUALITY-DIVERSITY OPTIMIZATION. Workshop on Agent Learning in Open-Endedness (ALOE) at ICLR 2022, 2022, virtual, Vatican City. ⟨hal-03753541⟩
41 Consultations
49 Téléchargements

Partager

Gmail Facebook X LinkedIn More