Communication Dans Un Congrès Année : 2024

Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization

Résumé

This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence.

Dates et versions

hal-04901527 , version 1 (20-01-2025)

Identifiants

Citer

Hanmin Li, Avetik Karagulyan, Peter Richtárik. Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization. International Conference on Learning Representations, May 2024, Vienna, Austria. ⟨hal-04901527⟩

Collections

28 Consultations
0 Téléchargements

Altmetric

Partager

  • More