Length Independent PAC-Bayes Bounds for Simple RNNs - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Length Independent PAC-Bayes Bounds for Simple RNNs

Résumé

While the practical interest of Recurrent Neural Networks (RNNs) is attested, much remains to be done to develop a thorough theoretical understanding of their abilities, particularly in what concerns their learning capacities. A powerful framework to tackle this question is the one of PAC-Bayes theory, which allows one to derive bounds providing guarantees on the expected performance of learning models on unseen data. In this paper, we provide an extensive study on the conditions leading to PAC-Bayes bounds for non-linear RNNs that are independent of the length of the data. The derivation of our results relies on a perturbation analysis on the weights of the network. We prove bounds that hold for β-saturated and DS β-saturated SRNs, classes of RNNs we introduce to formalize saturation regimes of RNNs. The first regime corresponds to the case where the values of the hidden state of the SRN are always close to the boundaries of the activation functions. The second one, closely related to practical observations, only requires that it happens at least once in each component of the hidden state on a sliding window of a given size.
Fichier principal
Vignette du fichier
PAC_Bayes_bound_for_SRNs__Camera_ready_.pdf (3.51 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-04488664 , version 1 (04-03-2024)

Identifiants

  • HAL Id : hal-04488664 , version 1

Citer

Volodimir Mitarchuk, Clara Lacroce, Rémi Eyraud, Rémi Emonet, Amaury Habrard, et al.. Length Independent PAC-Bayes Bounds for Simple RNNs. AISTATS 2024 - 27th International Conference on Artificial Intelligence and Statistics, May 2024, Valence, Spain. ⟨hal-04488664⟩
31 Consultations
15 Téléchargements

Partager

Gmail Facebook X LinkedIn More