A Shooting Formulation of Deep Learning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

A Shooting Formulation of Deep Learning

Résumé

Continuous-depth neural networks can be viewed as deep limits of discrete neural networks whose dynamics resemble a discretization of an ordinary differential equation (ODE). Although important steps have been taken to realize the advantages of such continuous formulations, most current techniques are not truly continuous-depth as they assume \textit{identical} layers. Indeed, existing works throw into relief the myriad difficulties presented by an infinite-dimensional parameter space in learning a continuous-depth neural ODE. To this end, we introduce a shooting formulation which shifts the perspective from parameterizing a network layer-by-layer to parameterizing over optimal networks described only by a set of initial conditions. For scalability, we propose a novel particle-ensemble parametrization which fully specifies the optimal weight trajectory of the continuous-depth neural network. Our experiments show that our particle-ensemble shooting formulation can achieve competitive performance, especially on long-range forecasting tasks. Finally, though the current work is inspired by continuous-depth neural networks, the particle-ensemble shooting formulation also applies to discrete-time networks and may lead to a new fertile area of research in deep learning parametrization.
Fichier non déposé

Dates et versions

hal-04497557 , version 1 (10-03-2024)

Identifiants

Citer

François-Xavier Vialard, Roland Kwitt, Susan Wei, Marc Niethammer. A Shooting Formulation of Deep Learning. NeurIPS 2020, 2020, Virtual conference, France. pp.11828-11838, ⟨10.48550/arXiv.2006.10330⟩. ⟨hal-04497557⟩
4 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More