Both white and brown adipose tissues are recognized to be differently involved in energy metabolism and are also able to secrete a variety of factors called adipokines that are involved in a wide range of physiological and metabolic functions. Brown adipose tissue is predominant around birth, except in pigs. Irrespective of species, white adipose tissue has a large capacity to expand postnatally and is able to adapt to a variety of factors. The aim of this review is to update the cellular and molecular mechanisms associated with pre- and postnatal adipose tissue development with a special focus on pigs and ruminants. In contrast to other tissues, the embryonic origin of adipose cells remains the subject of debate. Adipose cells arise from the recruitment of specific multipotent stem cells/progenitors named adipose tissue-derived stromal cells. Recent studies have highlighted the existence of a variety of those cells being able to differentiate into white, brown or brown-like/beige adipocytes. After commitment to the adipocyte lineage, progenitors undergo large changes in the expression of many genes involved in cell cycle arrest, lipid accumulation and secretory functions. Early nutrition can affect these processes during fetal and perinatal periods and can also influence or pre-determinate later growth of adipose tissue. How these changes may be related to adipose tissue functional maturity around birth and can influence newborn survival is discussed. Altogether, a better knowledge of fetal and postnatal adipose tissue development is important for various aspects of animal production, including neonatal survival, postnatal growth efficiency and health.