A Principle of Least Action for the Training of Neural Networks - Archive ouverte HAL Access content directly
Conference Papers Year : 2020

A Principle of Least Action for the Training of Neural Networks

Skander Karkar
Emmanuel de Bezenac
  • Function : Author
  • PersonId : 1101543
  • IdRef : 259426148

Abstract

Neural networks have been achieving high generalization performance on many tasks despite being highly over-parameterized. Since classical statistical learning theory struggles to explain this behaviour, much effort has recently been focused on uncovering the mechanisms behind it, in the hope of developing a more adequate theoretical framework and having a better control over the trained models. In this work, we adopt an alternative perspective, viewing the neural network as a dynamical system displacing input particles over time. We conduct a series of experiments and, by analyzing the network's behaviour through its displacements, we show the presence of a low kinetic energy bias in the transport map of the network, and link this bias with generalization performance. From this observation, we reformulate the learning problem as follows: find neural networks that solve the task while transporting the data as efficiently as possible. This offers a novel formulation of the learning problem which allows us to provide regularity results for the solution network, based on Optimal Transport theory. From a practical viewpoint, this allows us to propose a new learning algorithm, which automatically adapts to the complexity of the task, and leads to networks with a high generalization ability even in low data regimes.
Fichier principal
Vignette du fichier
2009.08372.pdf (932.08 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03038615 , version 1 (03-12-2020)

Identifiers

  • HAL Id : hal-03038615 , version 1

Cite

Skander Karkar, Ibrahim Ayed, Emmanuel de Bezenac, Patrick Gallinari. A Principle of Least Action for the Training of Neural Networks. ECML PKDD, Sep 2020, Ghent, Belgium. ⟨hal-03038615⟩
180 View
213 Download

Share

Gmail Facebook X LinkedIn More