Scaling by subsampling for big data, with applications to statistical learning - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Journal of Nonparametric Statistics Année : 2023

Scaling by subsampling for big data, with applications to statistical learning

Résumé

Handling large datasets and calculating complex statistics on huge datasets require important computing resources. Using subsampling methods to calculate statistics of interest on small samples is often used in practice to reduce computational complexity, for instance using the divide and conquer strategy. In this article, we recall some results on subsampling distributions and derive a precise rate of convergence for these quantities and the corresponding quantiles. We also develop some standardization techniques based on subsampling unstandardized statistics in the framework of large datasets. It is argued that using several subsampling distributions with different subsampling sizes brings a lot of information on the behavior of statistical learning procedures: subsampling allows to estimate the rate of convergence of different algorithms, to estimate the variability of complex statistics, to estimate confidence intervals for out-of-sample errors and interpolate their values at larger scales. These results are illustrated on simulations, but also on two important datasets, frequently analyzed in the statistical learning community, EMNIST (recognition of digits) and VeReMi (analysis of Network Vehicular Reference Misbehavior).
Fichier principal
Vignette du fichier
Scaling_by_subsampling_for_big_data__Bertail_et_al_JNPS.pdf (750.36 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04244852 , version 2 (16-10-2023)
hal-04244852 , version 1 (02-11-2023)

Identifiants

Citer

Patrice Bertail, Mohammed Bouchouia, Ons Jelassi, Jessica Tressou, Mélanie Zetlaoui. Scaling by subsampling for big data, with applications to statistical learning. Journal of Nonparametric Statistics, 2023, 36 (1), pp.78-117. ⟨10.1080/10485252.2023.2219782⟩. ⟨hal-04244852v2⟩
235 Consultations
56 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More