Fast and accurate optimization on the orthogonal manifold without retraction - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Fast and accurate optimization on the orthogonal manifold without retraction

Pierre Ablin

Résumé

We consider the problem of minimizing a function over the manifold of orthogonal matrices. The majority of algorithms for this problem compute a direction in the tangent space, and then use a retraction to move in that direction while staying on the manifold. Unfortunately, the numerical computation of retractions on the orthogonal manifold always involves some expensive linear algebra operation, such as matrix inversion, exponential or square-root. These operations quickly become expensive as the dimension of the matrices grows. To bypass this limitation, we propose the landing algorithm which does not use retractions. The algorithm is not constrained to stay on the manifold but its evolution is driven by a potential energy which progressively attracts it towards the manifold. One iteration of the landing algorithm only involves matrix multiplications, which makes it cheap compared to its retraction counterparts. We provide an analysis of the convergence of the algorithm, and demonstrate its promises on large-scale and deep learning problems, where it is faster and less prone to numerical errors than retraction-based methods.

Dates et versions

hal-03869218 , version 1 (24-11-2022)

Identifiants

Citer

Gabriel Peyré, Pierre Ablin. Fast and accurate optimization on the orthogonal manifold without retraction. 25th International Conference on Artificial Intelligence and Statistics, Mar 2022, Virtual, France. pp.5636-5657. ⟨hal-03869218⟩
11 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More