Ancient DNA studies reveal the genetic structure of Africa before the expansion of Bantu-speaking agriculturalists; however, the impact of now extinct hunter-gatherer and herder societies on the genetic makeup of present-day African groups remains elusive. Here, we uncover the genetic legacy of pre-Bantu populations from the Angolan Namib Desert, where we located small-scale groups associated with enigmatic forager traditions, as well as the last speakers of the Khoe-Kwadi family’s Kwadi branch. By applying an ancestry decomposition approach to genome-wide data from these and other African populations, we reconstructed the fine-scale histories of contact emerging from the migration of Khoe-Kwadi–speaking pastoralists and identified a deeply divergent ancestry, which is exclusively shared between groups from the Angolan Namib and adjacent areas of Namibia. The unique genetic heritage of the Namib peoples shows how modern DNA research targeting understudied regions of high ethnolinguistic diversity can complement ancient DNA studies in probing the deep genetic structure of the African continent.