Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019 - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

Zhengying Liu
  • Fonction : Auteur
  • PersonId : 1038060
Zhen Xu
  • Fonction : Auteur
Rongrong Ji
  • Fonction : Auteur
Ge Li
  • Fonction : Auteur
Jin Wang
  • Fonction : Auteur
Peng Wang
Yang Zhang

Résumé

The objective of this research is to push the frontiers in Automated Machine Learning, specifically targeting Deep Learning. We analyse ChaLearn's Automated Deep Learning challenge whose design features include: (i) Code submissions entirely blind-tested, on five classification problems during development, then ten others during final testing. (ii) Raw data from various modalities (image, video, text, speech, tabular data), formatted as tensors. (iii) Emphasis on "any-time learning" strategies by imposing fixed time/memory resources and using the Area under Learning curve as metric. (iv) Baselines provided, including "Baseline 3", combining top-ranked solutions of past rounds (AutoCV, AutoNLP, AutoSpeech,and AutoSeries). (v) No Deep Learning imposed. Principal findings: (1) The top two winners passed all final tests without failure, a significant step towards true automation. Their solutions were open-sourced. (2) Despite our effort to format all datasets uniformly to encourage generic solutions, the participants adopted specific workflows for each modality. (3) Anytime learning was addressed successfully, without sacrificing final performance. (4) Although some solutions improved over Baseline 3, it strongly influenced many. (5) Deep Learning solutions dominated, but Neural Architecture Search was impractical within the time budget imposed. Most solutions relied on fixed-architecture pre-trained networks, with fine-tuning. Ablation studies revealed the importance of meta-learning, ensembling, and efficient data loading, while data-augmentation is not critical.
Fichier principal
Vignette du fichier
autodl-tpami-no-heading-no-typesetting (1).pdf (997.69 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02957135 , version 1 (04-10-2020)
hal-02957135 , version 2 (30-11-2020)
hal-02957135 , version 3 (06-05-2021)
hal-02957135 , version 4 (01-12-2021)
hal-02957135 , version 5 (08-01-2022)

Identifiants

  • HAL Id : hal-02957135 , version 2

Citer

Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, et al.. Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019. 2020. ⟨hal-02957135v2⟩
406 Consultations
552 Téléchargements

Partager

Gmail Facebook X LinkedIn More