Orthogonal directions constrained gradient method: from non-linear equality constraints to stiefel manifold - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Orthogonal directions constrained gradient method: from non-linear equality constraints to stiefel manifold

Résumé

We consider the problem of minimizing a non-convex function over a smooth manifold $\mathcal{M}$. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards $\mathcal{M}$. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities $\mathcal{O}(1/\varepsilon^2)$ and $\mathcal{O}(1/\varepsilon^4)$ in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyr\'e (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyr\'e (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.

Dates et versions

hal-04273789 , version 1 (07-11-2023)

Identifiants

Citer

Sholom Schechtman, Daniil Tiapkin, Michael Muehlebach, Éric Moulines. Orthogonal directions constrained gradient method: from non-linear equality constraints to stiefel manifold. The 36th Annual Conference on Learning Theory (COLT 2023), Jul 2023, Bangalore, India. pp.1228--1258. ⟨hal-04273789⟩
35 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More