Deciphering and Optimizing Multi-Task Learning: a Random Matrix Approach - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Deciphering and Optimizing Multi-Task Learning: a Random Matrix Approach

Malik Tiomoko
  • Fonction : Auteur
  • PersonId : 1091598
Hafiz Tiomoko
  • Fonction : Auteur
  • PersonId : 1093104

Résumé

This article provides theoretical insights into the inner workings of multi-task and transfer learning methods, by studying the tractable least-square support vector machine multi-task learning (LS-SVM MTL) method, in the limit of large (p) and numerous (n) data. By a random matrix analysis applied to a Gaussian mixture data model, the performance of MTL LS-SVM is shown to converge, as n, p → ∞, to a deterministic limit involving simple (small-dimensional) statistics of the data. We prove (i) that the standard MTL LS-SVM algorithm is in general strongly biased and may dramatically fail (to the point that individual single-task LS-SVMs may outperform the MTL approach, even for quite resembling tasks): our analysis provides a simple method to correct these biases, and that we reveal (ii) the sufficient statistics at play in the method, which can be efficiently estimated, even for quite small datasets. The latter result is exploited to automatically optimize the hyperparameters without resorting to any cross-validation procedure. Experiments on popular datasets demonstrate that our improved MTL LS-SVM method is computationally-efficient and outperforms sometimes much more elaborate state-of-the-art multi-task and transfer learning techniques.
Fichier principal
Vignette du fichier
Transfer_Learning_for_ICLR_2021 (2).pdf (301.2 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03148622 , version 1 (22-02-2021)

Identifiants

  • HAL Id : hal-03148622 , version 1

Citer

Malik Tiomoko, Hafiz Tiomoko, Romain Couillet. Deciphering and Optimizing Multi-Task Learning: a Random Matrix Approach. ICLR 2021 - 9th International Conference on Learning Representations, May 2021, Vienna, Austria. ⟨hal-03148622⟩
134 Consultations
123 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More