Empirical Risk Minimization with Relative Entropy Regularization - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Information Theory Année : 2024

Empirical Risk Minimization with Relative Entropy Regularization

Résumé

The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a σ-finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, often mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERMRER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.
Fichier principal
Vignette du fichier
DoubleColumn.pdf (779.47 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03849748 , version 1 (12-11-2022)
hal-03849748 , version 2 (12-06-2023)
hal-03849748 , version 3 (29-11-2023)
hal-03849748 , version 4 (08-04-2024)

Identifiants

Citer

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Relative Entropy Regularization. IEEE Transactions on Information Theory, In press, pp.1-1. ⟨10.1109/TIT.2024.3365728⟩. ⟨hal-03849748v3⟩

Relations

148 Consultations
165 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More