The location and timing of domestication of the olive tree, a key crop in Early Mediterranean societies, remain hotly debated. Here, we unravel the history of wild olives (oleasters), and then infer the primary origins of the domesticated olive. Phylogeography and Bayesian molecular dating analyses based on plastid genome profiling of 1263 oleasters and 534 cultivated genotypes reveal three main lineages of pre-Quaternary origin. Regional hotspots of plastid diversity, species distribution modelling and macrofossils support the existence of three long-term refugia; namely the Near East (including Cyprus), the Aegean area and the Strait of Gibraltar. These ancestral wild gene pools have provided the essential foundations for cultivated olive breeding. Comparison of the geographical pattern of plastid diversity between wild and cultivated olives indicates the cradle of first domestication in the northern Levant followed by dispersals across the Mediterranean basin in parallel with the expansion of civilizations and human exchanges in this part of the world.