This whitepaper presents an in-depth exploration of Tabular Foundation Models (TFMs), an emerging area in the broader context of deep learning and Foundation Models. Foundation Models have been pivotal in driving technological advances in fields like natural language processing and computer vision, demonstrated by groundbreaking models such as GPT or DALL-E. Their success, primarily attributed to scale and emergent properties, has enabled capabilities like zero-shot learning and extensive generalization. However, the extension of these models to tabular data, which is both abundant and critical in numerous practical applications, has only recently begun to attract significant attention. We briefly introduce the concept and evolution of Foundation Models, highlighting their successes in various domains and the key factors contributing to their performance. Subsequently, we discuss the state of the art in tabular deep learning in detail, highlighting recent representation learning approaches. Centrally, we discussthe technical challenges arising in cross-table representation learning - a key towards developing Tabular Foundation Models. These include the unique characteristics of tabular data, such as heterogeneity and the need for specialized handling in tokenization, which differentiate it from other data types. Finally, we propose a preliminary architecture and training pipeline as a baseline towards building Tabular Foundation Models, providing initial experimental results. This serves as a starting point for further research and development in this promising area.