At present, a strong latitudinal sea surface temperature (SST) gradient of ~16°C exists across the Southern Ocean, maintained by the Antarctic Circumpolar Current (ACC) and a set of complex frontal systems. Together with the Antarctic ice masses, this system has formed one of the most important global climate regulators. The timing of the onset of the ACCsystem, its development towards modern-day strength, and the consequences for e.g., the latitudinal SST gradient around the southern Atlantic Ocean, are still uncertain. Here we present new TEX$_{86}$-biomarker records, calibrated to SST, from two sites located east of Drake Passage (southern South Atlantic) to assist in better understanding two critical time intervals of prominent climate transitions during the Cenozoic: The Late Eocene-Early Oligocene (ODP Site 696) and Middle-Late Miocene (IODP Site U1536) transitions. Our results overall show rather temperate conditions (20-11°C) during the Late Eocene to Early Oligocene interval, with a weaker latitudinal SST gradient (~8°C) across the Atlantic sector of the Southern Ocean compared to present day (~16°C). We ascribe the regional similarity in SSTs across the Late Eocene-Early Oligocene South Atlantic to a persistent, strong Subpolar Gyre circulation, connecting all sites, which can only exist in absence of a strong throughflow across the Drake Passage. Surprisingly, the southern South Atlantic records show comparable SSTs (~12-14°C) during both the Earliest Oligocene Oxygen Isotope Step (EOIS, ~33.65 Ma) and the Miocene Climate Optimum (MCO, ~16.5 Ma). Apparently, maximum Oligocene Antarctic ice volume could coexist with warm ice-proximal surface ocean conditions, while at similar ocean temperatures, the Middle Miocene Antarctic ice sheet was strongly reduced. Southern South Atlantic SSTs cooled to ~5°C at the onset of the Middle Miocene Climate Transition (MMCT, 14 Ma), making it the coldest oceanic region recorded around Antarctica and the likely main location for deep water formation. The already cold southern South Atlantic conditions at MMCT meant it experienced little cooling during the latter part of the Miocene, which contrasts the profound cooling due to northward expansion of the Southern Ocean frontal systems in the lower latitudes and other sectors of the Southern Ocean.