The deep history of humans in Africa and the complex divergences and migrations among ancient human genetic lineages remain poorly understood and are the subject of ongoing debate. We produced 73 high-quality whole genome sequences from 14 Central and Southern African populations with diverse, well-documented, languages, subsistence strategies, and socio-cultural practices, and jointly analyze this novel data with 104 African and non-African previously-released whole genomes. We find vast genome-wide diversity and individual pairwise differentiation within and among African populations at continental, regional, and even local geographical scales, often uncorrelated with linguistic affiliations and cultural practices. We combine populations in 54 different ways and, for each population combination separately, we conduct extensive machine-learning Approximate Bayesian Computation inferences relying on genome-wide simulations of 48 competing evolutionary scenarios. We thus reconstruct jointly the tree-topologies and migration processes among ancient and recent lineages best explaining the diversity of extant genomic patterns. Our results show the necessity to explicitly consider the genomic diversity of African populations at a local scale, without merging population samples indiscriminately into larger a priori categories based on geography, subsistence-strategy, and/or linguistics criteria, in order to reconstruct the diverse evolutionary histories of our species. We find that, for all different combinations of Central and Southern African populations, a tree-like evolution with long periods of drift between short periods of unidirectional gene-flow among pairs of ancient or recent lineages best explain observed genomic patterns compared to recurring gene-flow processes among lineages. Moreover, we find that, for 25 combinations of populations, the lineage ancestral to extant Southern African Khoe-San populations diverged around 300,000 years ago from a lineage ancestral to Rainforest Hunter-Gatherers and neighboring agriculturalist populations. We also find that short periods of ancient or recent asymmetrical gene-flow among lineages often coincided with epochs of major cultural and ecological changes previously identified by paleo-climatologists and archaeologists in Sub-Saharan Africa.