On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations

Résumé

Stochastic optimization methods encounter new challenges in the realm of streaming, characterized by a continuous flow of large, high-dimensional data. While first-order methods, like stochastic gradient descent, are the natural choice, they often struggle with ill-conditioned problems. In contrast, second-order methods, such as Newton's methods, offer a potential solution, but their computational demands render them impractical. This paper introduces adaptive stochastic optimization methods that bridge the gap between addressing ill-conditioned problems while functioning in a streaming context. Notably, we present an adaptive inversion-free Newton's method with a computational complexity matching that of first-order methods, $\mathcal{O}(dN)$, where $d$ represents the number of dimensions/features, and $N$ the number of data. Theoretical analysis confirms their asymptotic efficiency, and empirical evidence demonstrates their effectiveness, especially in scenarios involving complex covariance structures and challenging initializations. In particular, our adaptive Newton's methods outperform existing methods, while maintaining favorable computational efficiency.
Fichier principal
Vignette du fichier
main.pdf (787.68 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04308712 , version 1 (27-11-2023)
hal-04308712 , version 2 (30-01-2024)

Identifiants

Citer

Antoine Godichon-Baggioni, Nicklas Werge. On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations. 2024. ⟨hal-04308712v2⟩
56 Consultations
9 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More