Data-dependent Generalization Bounds via Variable-Size Compressibility - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Information Theory Année : 2024

Data-dependent Generalization Bounds via Variable-Size Compressibility

Milad Sefidgaran

Résumé

In this paper, we establish novel data-dependent upper bounds on the generalization error through the lens of a "variable-size compressibility" framework that we introduce newly here. In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data. This is shown to yield bounds that depend on the empirical measure of the given input data at hand, rather than its unknown distribution. Our new generalization bounds that we establish are tail bounds, tail bounds on the expectation, and in-expectations bounds. Moreover, it is shown that our framework also allows to derive general bounds on any function of the input data and output hypothesis random variables. In particular, these general bounds are shown to subsume and possibly improve over several existing PAC-Bayes and data-dependent intrinsic dimension-based bounds that are recovered as special cases, thus unveiling a unifying character of our approach. For instance, a new data-dependent intrinsic dimension-based bound is established, which connects the generalization error to the optimization trajectories and reveals various interesting connections with the rate-distortion dimension of a process, the R\'enyi information dimension of a process, and the metric mean dimension.

Dates et versions

hal-04457385 , version 1 (14-02-2024)

Identifiants

Citer

Milad Sefidgaran, Abdellatif Zaidi. Data-dependent Generalization Bounds via Variable-Size Compressibility. IEEE Transactions on Information Theory, In press. ⟨hal-04457385⟩
32 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More