Pré-Publication, Document De Travail Année : 2024

Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines

Résumé

Restricted Boltzmann machines (RBMs) are among the simplest unsupervised models implementing data/representation duality. The learning curves of RBMs trained on structured data are nevertheless difficult to characterize analytically, in part due to the presence of a partition function that depends on the trainable parameters. In this work, we present the exact solution of RBMs trained on structured data in the undersampled regime. The solution involves gradual symmetry breaking among the hidden units for decreasing regularization strength, as they specialize to finer-level details of the data. Hidden units form extensive blocks with identical weight parameters. Trained RBMs with different block sizes are separated by large barriers in the posterior distribution of the weights, which makes the optimal block size inaccessible during training with local gradient descent.

Fichier principal
Vignette du fichier
2024_Undersampled__RBM.pdf (527.37 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04447899 , version 1 (08-02-2024)

Licence

Identifiants

  • HAL Id : hal-04447899 , version 1

Citer

Jorge Fernandez-De-Cossio-Diaz, Thomas Tulinski, Simona Cocco, Rémi Monasson. Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines. 2024. ⟨hal-04447899⟩
497 Consultations
835 Téléchargements

Partager

  • More