Article Dans Une Revue IMA Journal of Numerical Analysis Année : 2025

Deterministic and probabilistic rounding error analysis of neural networks in floating-point arithmetic

Résumé

The use of artificial neural networks is now becoming widespread across a wide variety of tasks. In this context of very rapid development, issues related to the storage and computational performance of these models emerge, since networks are sometimes very deep and comprise up to trillions of parameters. For all these reasons, the use of reduced precision is increasingly being considered although, until now, its accuracy and robustness had been approached mostly from a practical standpoint or verified by software. The aim of this work is to provide formal tools to better understand, explain, and predict the accuracy and stability of neural networks when using floating-point arithmetic. To this end, we apply a rounding error analysis based on existing tools in numerical linear algebra to obtain both forward and backward error bounds. This includes both deterministic worst-case bounds as well as probabilistic bounds that are sharper on average. Since the exact backward error is not directly computable, the backward error bounds are validated experimentally through a linearized backward error. These bounds both ensure the proper functioning of neural networks once trained, and provide recommendations on architectures and training methods to enhance the robustness of neural networks.

Fichier principal
Vignette du fichier
deterministic_probabilistic_rounding_error_analysis_nn.pdf (526.5 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04663142 , version 1 (26-07-2024)
hal-04663142 , version 2 (07-10-2025)
hal-04663142 , version 3 (09-02-2026)

Licence

Identifiants

  • HAL Id : hal-04663142 , version 3

Citer

Théo Beuzeville, Alfredo Buttari, Serge Gratton, Theo Mary. Deterministic and probabilistic rounding error analysis of neural networks in floating-point arithmetic. IMA Journal of Numerical Analysis, inPress. ⟨hal-04663142v3⟩
12824 Consultations
1669 Téléchargements

Partager

  • More