Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail (Preprint/Prepublication) Année : 2024

Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

Résumé

The solution to empirical risk minimization with f-divergence regularization (ERM-fDR) is presented under mild conditions on f. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function f are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of f-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of f-divergences when used in the ERM-fDR problem: i) f-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and ii) any f-divergence regularization is equivalent to a different f-divergence regularization with an appropriate transformation of the empirical risk function.
Fichier principal
Vignette du fichier
ISIT_2024_ERM_with_fDR.pdf (339.71 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-04431558 , version 1 (01-02-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04431558 , version 1

Citer

Francisco Daunas, Iñaki Esnaola, Samir M Perlaza, H. Vincent Poor. Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences. 2024. ⟨hal-04431558⟩
52 Consultations
21 Téléchargements

Partager

Gmail Facebook X LinkedIn More