First-order and second-order variants of the gradient descent in a unified framework - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2019

First-order and second-order variants of the gradient descent in a unified framework

Résumé

In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the classical and generalized Gauss-Newton methods, the natural gradient descent method, the gradient covariance matrix approach, and Newton's method. Besides interpreting these methods within a single framework, we explain their specificities and show under which conditions some of them coincide.

Dates et versions

hal-02397757 , version 1 (06-12-2019)

Identifiants

Citer

Thomas Pierrot, Nicolas Perrin, Olivier Sigaud. First-order and second-order variants of the gradient descent in a unified framework. 2019. ⟨hal-02397757⟩
56 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More