Towards Tabular Foundation Models - Archive ouverte HAL Accéder directement au contenu
Rapport (Rapport De Recherche) Année : 2024

Towards Tabular Foundation Models

Résumé

This whitepaper presents an in-depth exploration of Tabular Foundation Models (TFMs), an emerging area in the broader context of deep learning and Foundation Models. Foundation Models have been pivotal in driving technological advances in fields like natural language processing and computer vision, demonstrated by groundbreaking models such as GPT or DALL-E. Their success, primarily attributed to scale and emergent properties, has enabled capabilities like zero-shot learning and extensive generalization. However, the extension of these models to tabular data, which is both abundant and critical in numerous practical applications, has only recently begun to attract significant attention. We briefly introduce the concept and evolution of Foundation Models, highlighting their successes in various domains and the key factors contributing to their performance. Subsequently, we discuss the state of the art in tabular deep learning in detail, highlighting recent representation learning approaches. Centrally, we discussthe technical challenges arising in cross-table representation learning - a key towards developing Tabular Foundation Models. These include the unique characteristics of tabular data, such as heterogeneity and the need for specialized handling in tokenization, which differentiate it from other data types. Finally, we propose a preliminary architecture and training pipeline as a baseline towards building Tabular Foundation Models, providing initial experimental results. This serves as a starting point for further research and development in this promising area.
Fichier principal
Vignette du fichier
2024_Tabular_Foundation_Models_Whitepaper.pdf (1.02 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04440710 , version 1 (06-02-2024)

Licence

Identifiants

  • HAL Id : hal-04440710 , version 1

Citer

Maximilian Schambach. Towards Tabular Foundation Models: Status quo, challenges, and opportunities. Merantix Momentum. 2024. ⟨hal-04440710⟩

Collections

LARA
316 Consultations
402 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More