Communication Dans Un Congrès Année : 2025

Learning Permutations in Monarch Factorization

Résumé

In order to reduce the quadratic cost of matrixvector multiplications in dense and attention layers, Monarch matrices have been recently introduced, achieving a sub-quadratic complexity. It consists in factorizing a matrix using fixed permutations and learned block diagonal matrices, at the price of a small performance drop. We propose a more general model where some permutations are learned. The optimization algorithm explores the space of permutations using a Straight-Through Estimator (STE) inspired by the support exploration algorithm designed for sparse support recovery. Our experimental results demonstrate performance improvement in the context of sparse matrix factorization and of end-to-end sparse learning.

Fichier principal
Vignette du fichier
ICASSP_2025_Monarch.pdf (317.83 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04887483 , version 1 (17-01-2025)

Licence

Identifiants

Citer

Mimoun Mohamed, Valentin Emiya, Caroline Chaux. Learning Permutations in Monarch Factorization. ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Apr 2025, Hyderabad, India. ⟨10.1109/ICASSP49660.2025.10889798⟩. ⟨hal-04887483⟩
519 Consultations
554 Téléchargements

Altmetric

Partager

  • More