Is My Neural Net Driven by the MDL Principle? - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

Is My Neural Net Driven by the MDL Principle?

Eduardo Brandao
Rémi Emonet
Amaury Habrard
François Jacquenet
Marc Sebban

Résumé

The Minimum Description Length principle (MDL) is a formalization of Occam's razor for model selection, which states that a good model is one that can losslessly compress the data while including the cost of describing the model itself. While MDL can naturally express the behavior of certain models such as autoencoders (that inherently compress data) most representation learning techniques do not rely on such models. Instead, they learn representations by training on general or, for self-supervised learning, pretext tasks. In this paper, we propose a new formulation of the MDL principle that relies on the concept of signal and noise, which are implicitly defined by the learning task at hand. Additionally, we introduce ways to empirically measure the complexity of the learned representations by analyzing the spectra of the point Jacobians. Under certain assumptions, we show that the singular values of the point Jacobians of Neural Networks driven by the MDL principle should follow either a power law or a lognormal distribution. Finally, we conduct experiments to evaluate the behavior of the proposed measure applied to deep neural networks on different datasets, with respect to several types of noise. We observe that the experimental spectral distribution is in agreement with the spectral distribution predicted by our MDL principle, which suggests that neural networks trained with gradient descent on noisy data implicitly abide the MDL principle.
Fichier principal
Vignette du fichier
Eduardo_ECML_2023__Copy_-7.pdf (13.46 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04231405 , version 1 (06-10-2023)

Identifiants

Citer

Eduardo Brandao, Stefan Duffner, Rémi Emonet, Amaury Habrard, François Jacquenet, et al.. Is My Neural Net Driven by the MDL Principle?. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Sep 2023, Turin, France. pp.173-189, ⟨10.1007/978-3-031-43415-0_11⟩. ⟨hal-04231405⟩
92 Consultations
108 Téléchargements

Altmetric

Partager

More