Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines

Résumé

Restricted Boltzmann machines (RBMs) are among the simplest unsupervised models implementing data/representation duality. The learning curves of RBMs trained on structured data are nevertheless difficult to characterize analytically, in part due to the presence of a partition function that depends on the trainable parameters. In this work, we present the exact solution of RBMs trained on structured data in the undersampled regime. The solution involves gradual symmetry breaking among the hidden units for decreasing regularization strength, as they specialize to finer-level details of the data. Hidden units form extensive blocks with identical weight parameters. Trained RBMs with different block sizes are separated by large barriers in the posterior distribution of the weights, which makes the optimal block size inaccessible during training with local gradient descent.
Fichier principal
Vignette du fichier
2024_Undersampled__RBM.pdf (527.37 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04447899 , version 1 (08-02-2024)

Identifiants

  • HAL Id : hal-04447899 , version 1

Citer

Jorge Fernandez-De-Cossio-Diaz, Thomas Tulinski, Simona Cocco, Rémi Monasson. Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines. 2024. ⟨hal-04447899⟩
154 Consultations
172 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More