Replica symmetry breaking and clustering phase transitions in undersampled restricted Boltzmann machines
Résumé
Restricted Boltzmann machines (RBMs) are among the simplest unsupervised models implementing data/representation duality. The learning curves of RBMs trained on structured data are nevertheless difficult to characterize analytically, in part due to the presence of a partition function that depends on the trainable parameters. In this work, we present the exact solution of RBMs trained on structured data in the undersampled regime. The solution involves gradual symmetry breaking among the hidden units for decreasing regularization strength, as they specialize to finer-level details of the data. Hidden units form extensive blocks with identical weight parameters. Trained RBMs with different block sizes are separated by large barriers in the posterior distribution of the weights, which makes the optimal block size inaccessible during training with local gradient descent.
Origine | Fichiers produits par l'(les) auteur(s) |
---|